Add details long-context extrapolation

README.md

@@ -70,11 +70,12 @@ Rnj-1 is a family of 8B parameter open-weight, dense models trained from scratch
 
 # Changelog
 
-* Update December 18, 2025:
+* Update December 20, 2025:
   - System prompt and temperature recommendations: Resolve premature truncations and mitigate unprompted code outputs.
   - Updates to default chat template.
-  - Updated STEM and comparables tables.
+  - Updated evaluation results.
   - Links to model generations for evals.
+  - Instructions for long-context extrapolation.
 
 * Initial version: December 8, 2025
 
@@ -169,6 +170,50 @@ The global batch sizes used were:
 - 24M tokens for mid-training.
 - 16M tokens for SFT.
 
+### Long-Context Extrapolation (up to 128k)
+Although Rnj-1-Instruct was trained with context lengths up to 32k, the model can be extrapolated to 128k context using YaRN RoPE scaling. This requires the following updates to `config.json`:
+```diff
+@@
+-  "max_position_embeddings": 32768,
++  "max_position_embeddings": 131072,
+
+@@
+-  "sliding_window": 32768,
++  "sliding_window": 131072,
+
+@@
+   "rope_scaling": {
+     "attn_factor": 1.0,
+     "beta_fast": 64.0,
+     "beta_slow": 1.0,
+     "extrapolation_factor": 1.0,
+-    "factor": 4.0,
++    "factor": 16.0,
+     "original_max_position_embeddings": 8192,
+     "rope_type": "yarn"
+   },
+```
+
+Overall, most capabilities are preserved under 128k extrapolation, with performance remaining stable on many coding, math, SWE and FIM benchmarks. However, we do observe select regressions, particularly on some science and performance-based evaluations.
+
+| Category   | Evals                 | Rnj-1-instruct | Rnj-1-instruct (128k) |
+|------------|-----------------------|-------|--------------|
+| Coding     | MBPP+                 | 75.7  | 75.7         |
+| Coding     | HE+                   | 83.5  | 82.3         |
+| Coding     | BigCodeBench-full     | 57.1  | 55.3         |
+| Math       | AIME 25               | 43.3  | 53.3         |
+| Math       | GSM8k                 | 92.6  | 91.1         |
+| Math       | Minerva-MATH-500      | 88.4  | 89.4         |
+| Science    | MMLU-STEM             | 81.8  | 69.4         |
+| Science    | GPQA-Diamond          | 38.9  | 41.4         |
+| Env evals  | SWE-bench (bash)      | 20.8  | 20.1         |
+| Env evals  | Performance: Enamel   | 49.0  | 39.9         |
+| FIM        | HE single-line        | 94.9  | 93.5         |
+| FIM        | HE multi-line         | 77.6  | 76.5         |
+| FIM        | HE random-span        | 86.1  | 85.1         |
+
+We are actively investigating mitigations (including improved scaling strategies and targeted long-context tuning) and expect to close much of this gap in future updates.
+
 # Recommendations
 
 ### System Prompt & Temperature