Episode Details

Researchers introduce TailSFT, a modified supervised fine-tuning algorithm designed to better prepare language models for subsequent reinforcement learning. Unlike standard fine-tuning that minimizes overall cross-entropy, TailSFT filters out sequences that the model has already mastered to focus training on the under-modeled "tail" of the data distribution. This approach prioritizes coverage, ensuring the model retains a diverse range of correct responses that reinforcement learning can later identify and amplify. Theoretical analysis and experiments on the OLMo-3 7B model demonstrate that TailSFT significantly boosts performance in math and coding tasks, particularly by improving pass@K metrics. Ultimately, the authors show that a higher-coverage initialization leads to faster learning and superior final accuracy after reinforcement learning. This work advocates for a stage-aware approach to AI development, where intermediate training phases are optimized specifically to benefit the next stage of the pipeline.