This website is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License. Theme by Nerfies.
Key Findings
Pre-training ▾
Where Do Returns Diminish?
We rigorously benchmark LMs of different sizes, increasing pre-training tokens far beyond traditional recipes (Chinchilla optimal ratio of 20x model size). As visualized in Figure 1, we track accuracy improvements at every token budget. While adding more tokens at first yields clear improvements, after a certain threshold, extra pre-training becomes less cost-effective.
Takeaway 1. Excessive general-domain pre-training improves upstream performance but with diminishing returns (saturation happens around 80x to 160x model size in our study).
When More Isn't Better: Downstream Surprises
Does endlessly scaling pre-training always help with downstream tasks? In Figure 2, we evaluate how different pre-training regimes affect real-world downstream performance, both for tasks similar to the mid-training and post-training data and for novel (OOD) tasks. Remarkably, overly pre-training does not always improve or can even harm downstream reasoning.
Takeaway 2. Excessive general-domain pre-training does not always improve domain-specific post-training and might even cause performance degradation on some downstream tasks (saturation happens around 80x to 160x model size in our study).
Small vs. Large Models: The Budget-Compute Tradeoff
A common assumption is that larger models will always outperform their smaller counterparts. We study whether this assumption holds under limited pre-training resources. We directly compare 1B and 4B models at fixed compute and data limits (Table 2), examining their downstream results. For limited resources, a well-tuned small model may be more effective, while larger models perform better only after a certain data threshold is met.
Takeaway 3. Under limited pre-training budgets, smaller post-trained models can even outperform larger counterparts. Conversely, once pre-training tokens reach the saturation regime, increasing model size enables clear improvements in both in-domain performance and OOD generalization.
Mid-training ▾
Catastrophic Forgetting in Continued Pre-training
How do models adapt to new domains without forgetting old knowledge? We investigate continued pre-training (CPT) with and without "replay" of general-domain data, tracking upstream performance in Figure 3 and downstream performance in Table 1. A small percentage of general replay (just 5%) proved critical for balancing new skills with retention of broad knowledge—an easy but powerful trick for practical domain adaptation.
Takeaway 4. Continued pre-training on domain-specific data induces catastrophic forgetting of pre-trained knowledge which could harm both upstream and downstream performance, while incorporating a small replay budget (e.g. 5%) could effectively mitigate this degradation.
Importance of Continued Pre-training for Post-training
We test the effect of varying CPT data volume on downstream results, showing results in Figure 4.
Inadequate domain data risks leaving the model poorly adapted, even after SFT or RL. Investing in rich domain datasets, on the other hand, is essential for strong post-training results.
Takeaway 5. Domain-specific post-training should be supported by adequate domain-specific CPT data: without it, SFT performance remains suboptimal and RL can even degrade such performance.
Our study reveals the upward trend in in-domain accuracy as more CPT tokens are used. This sustained improvement justifies larger domain-specific datasets, especially when downstream reasoning or RL enhancement is desired.
Takeaway 6. As domain-specific CPT data increase, in-domain downstream performance steadily improves and the SFT models could benefit more from RL finetuning.
We analyze the effect of high-volume CPT on both in-domain and OOD tasks. Well-designed domain adaptation can create more flexible models—not just specialists—by strengthening transferable reasoning abilities to OOD tasks.
Takeaway 7. With sufficient domain-specific CPT data, post-training on in-domain tasks not only improves in-domain performance but also generalizes effectively to OOD tasks.
Post-training ▾
SFT: Diminishing Returns and Overfitting Risks
We vary both SFT epochs and dataset size, charting downstream metrics in Figures 5 and 6, and show that more SFT is not always better. Overfitting is real, and can hurt generalization—fine-tuning should be done with care and strong validation.
Takeaway 8. Excessive SFT improves ID performance with diminishing returns but does not necessarily improve and can even degrade OOD performance.
By systematically increasing SFT before RL, we measure the headroom left for further RL gains. When a model is already over-specialized from SFT, RL has little left to improve. Keeping SFT at a balanced level leaves more opportunity for RL to make a difference.
Takeaway 9. Excessive SFT, especially overly large epochs, could limit further RL improvements.
RL: Diminishing Returns and Practical Solutions
We scale RL epochs and data size (see Figure 7), documenting how performance changes across different regimes. For both ID and OOD tasks, most of RL's benefit comes early. Targeting 4–8 epochs or ~100K examples gives a practical balance between results and cost in our study on 1B models.
Takeaway 10. RL with excessive epochs or examples improves downstream performance on both ID and OOD tasks but with diminishing returns (saturation happens at 4-8 epochs or 50-100K examples in our study).
Does RL make the model reason better, or just sample more confidently? We delve into RL's effect on solution diversity and quality. We show that after saturation, RL mainly sharpens the output distribution—helping you sample correct answers more often, but does not fundamentally improve reasoning skill—solving problems that cannot be solved.
Takeaway 11. Beyond saturation regime, RL primarily increases the probability of sampling high-quality rollouts but does not necessarily improve models' fundamental reasoning capabilities.
SFT/RL Allocation Under Data Constraints
Given a limited downstream budget, should you spend it on SFT or RL? We experiment with different SFT/RL splits (see Figure 8) on 1B and 4B models to quantify the trade-off between in-domain and OOD performance. Choose the allocation based on goals: favor SFT for specialists, or RL for generalists. This helps tailor the LM for the tasks that matter most.
Takeaway 12. Under a constrained downstream data budget, allocating more examples to SFT maximizes in-domain gains at the expense of weaker OOD generalization, while allocating more to RL improves OOD performance.
Validation ▾
ORM Score as Validation Metric
We assess ORM (Outcome Reward Model) scores on their ability to predict success on downstream tasks. ORM scoring gives a true picture of reasoning quality post-training. This can help researchers and engineers more reliably monitor and optimize their models during post-training, especially when a validation set (with ground truth labels) is not available or too expensive to collect.
Takeaway 13. ORM score could be a more reliable unsupervised validation metric that helps predict downstream task performance during post-training, compared to validation loss. Notably, ORM scores from an 8B reward model correlate well with problem-solving accuracies of 1B models on many downstream reasoning tasks.