Randomized YaRN Enhances Long-Context Reasoning in Large Language Models

Researchers have introduced a new training method called Randomized YaRN to improve length generalization in large language models (LLMs) for long-context reasoning tasks. Traditional LLMs, which are pretrained on short sequences, often struggle with very long sequences. Randomized YaRN combines YaRN-based positional extrapolation with randomized positional encoding and a length curriculum, allowing models to learn from a wider range of positional representations during training. The method was evaluated on two benchmarks, BABILong and Multi-Round Coreference Resolution (MRCR), demonstrating consistent improvements in reasoning performance for context lengths ranging from 16K to 128K, particularly when trained on data with less than 8K context. These findings indicate that exposing models to out-of-distribution positional distributions can significantly enhance long-context reasoning capabilities.