Researchers at Nvidia have developed a brand new approach that flips the script on how giant language fashions (LLMs) be taught to cause.
The tactic, referred to as reinforcement studying pre-training (RLP), integrates RL into the preliminary coaching part moderately than saving it for the tip.
This strategy encourages the mannequin to “assume for itself earlier than predicting what comes subsequent, thus instructing an unbiased pondering habits earlier within the pretraining,” the researchers state of their paper.
By studying to cause on plain textual content while not having exterior verifiers, fashions educated with RLP present important enhancements in studying advanced reasoning duties downstream, hinting at a way forward for extra succesful and adaptable AI for real-world duties.
The everyday LLM coaching cycle
Usually, giant language fashions are first pre-trained on huge quantities of textual content utilizing a "next-token prediction" goal, the place they’re given a string of textual content and requested to repeatedly guess what the following phrase (or token) can be. On this part, they be taught grammar, info, and primary associations.
Within the later post-training part, fashions normally be taught advanced reasoning skills reminiscent of chain-of-thought (CoT) the place a mannequin lays out its reasoning step-by-step. This stage usually includes supervised fine-tuning (SFT) or reinforcement studying from human suggestions (RLHF), which require specialised, curated datasets.
The paper’s authors argue this sequential course of doesn’t match human comprehension, which is “not a linear token-by-token course of, however moderately a parallel integration of enter with prior data.” Present pre-training strategies lack this mechanism, hindering a mannequin's capacity to develop deep reasoning from the beginning.
How reinforcement studying pre-training works
RLP reframes this course of by treating CoT technology as an motion the mannequin takes earlier than predicting the following token. At every step, the mannequin first generates an inner "thought" or reasoning chain. It then predicts the following phrase within the textual content, utilizing the unique context augmented with its new thought.
The mannequin receives a reward primarily based on how a lot its thought improved the accuracy of its prediction in comparison with a baseline that didn't generate a thought (pure next-token prediction). This reward sign is calculated routinely primarily based on the change in likelihood, eliminating the necessity for exterior verifiers or human-labeled knowledge.
The reward is optimistic solely when the generated thought helps the mannequin higher predict the following token. By rewarding ideas primarily based on their predictive profit, RLP successfully teaches the mannequin learn how to assume usefully on the identical large, unstructured datasets used for normal pre-training.
This steady suggestions loop permits the mannequin to be taught when a easy predictive guess is ample and when it wants to have interaction in deeper reasoning. Because the researchers put it, “RLP is designed to form pondering in base fashions by rewarding solely these ideas that measurably assist next-token prediction.”
This foundational strategy, nonetheless, doesn't make later fine-tuning phases out of date. Based on Bryan Catanzaro, VP of utilized deep studying analysis at Nvidia and a co-author of the paper, RLP is designed to enrich, not change, these essential steps. "RLP isn’t meant to exchange the later post-training phases like supervised fine-tuning or reinforcement studying from human suggestions," Catanzaro informed VentureBeat. "These phases stay essential for refining mannequin habits… It’s actually designed to amplify the effectiveness of these later phases by giving the mannequin a head begin."
RLP in motion
In experiments with Qwen3-1.7B and Nemotron-Nano-12B, Nvidia’s group examined RLP throughout a set of math and science reasoning benchmarks. The outcomes present that fashions enhanced with RLP constantly outperformed their conventionally educated counterparts, with significantly robust good points in reasoning-heavy duties.
For an enterprise, this improved reasoning may translate to extra dependable outputs in multi-step workflows like monetary evaluation or authorized doc summarization.
"RLP encourages the mannequin throughout pretraining to assume earlier than it predicts, serving to the mannequin internalize a extra coherent reasoning fashion," stated Catanzaro. "This might assist scale back delicate logical errors, particularly in longer workflows.”
Whereas stressing that RLP-trained fashions will nonetheless want the standard guardrails reminiscent of verification layers, human oversight, and consistency checks, Catanzaro stated that “RLP provides you a stronger baseline."
Importantly, the advantages of RLP compound as a substitute of disappearing throughout subsequent fine-tuning phases (catastrophic forgetting is a standard drawback in LLM coaching, the place later coaching phases trigger the mannequin to neglect its beforehand discovered expertise and data). The RLP-trained mannequin achieved an general rating that was 7-8% greater than baselines after an an identical post-training routine. The researchers conclude that RLP “establishes sturdy reasoning foundations that aren’t washed out by downstream alignment however as a substitute compound with post-training.”
The effectivity of the approach is a key discovering. On the Qwen3-1.7B mannequin, RLP improved efficiency by 17% over customary steady pre-training and in addition beat an identical approach referred to as Reinforcement Pretraining by way of prefix-matching rewards (RPT). This benefit held even when the baseline mannequin was educated with 35 occasions extra knowledge to match the computational value, confirming the good points come from the tactic itself, not simply extra processing.
Moreover, RLP demonstrates spectacular scalability and flexibility, efficiently extracting a reasoning sign from general-purpose internet knowledge—not simply curated datasets. When utilized to the hybrid Mamba-Transformer mannequin Nemotron-Nano-12B, RLP achieved a 35% relative enchancment over a closely educated baseline whereas utilizing only a tiny fraction of the info.
Whereas these outcomes level towards a extra environment friendly path for constructing highly effective fashions, Catanzaro frames the innovation as a basic shift within the studying course of itself, moderately than a right away answer to excessive coaching prices.
"This analysis is thrilling as a result of it provides a shift in how fashions take in data throughout pretraining resulting in a wiser studying course of," he defined. "It wouldn’t change large-scale pretraining, however supply one other inventive technique in constructing the very best fashions."
A brand new basis for AI coaching
In the end, RLP factors towards a future the place pre-training is not a monolithic technique of next-token prediction. As an alternative, the following technology of fashions might be constructed on a hybrid of targets, creating AI that learns to assume extra robustly from day one. Catanzaro provides a robust analogy to border this shift:
"Subsequent-token prediction teaches a mannequin what the world appears like; reinforcement-style targets like RLP can educate it how to consider what it’s seeing," he stated. "The mixture of those two targets may assist fashions develop deeper, extra structured pondering a lot earlier in coaching… Instruments like RLP can construct on high of that basis, making studying extra lively, curious, and much more environment friendly."
There’s nonetheless rather a lot to be taught in regards to the dynamics of reinforcement studying within the pre-training part, however what appears clear is that “introducing exploration earlier in coaching opens a brand new axis for scaling — not simply in dimension, however in how fashions be taught to cause,” Catanzaro stated.
