How can LLM RL Work Despite Information-Theoretic Inefficiency

Epistemic Status: Obviously speculative and maybe obvious.

The success of RL in LLMs has been puzzling me for a while. People have developed various information-theoretic style arguments by which they argue that RL is extremely informationally inefficient compared to pretraining, that it can only impart a tiny amount of bits, and that it can only bring out behaviours that are already in the base models etc. The basic intuition here is extremely obvious and essentially falls immediately out of the formulation of the two methods. Pretraining (and SFT etc) compute a loss on every token so every token gives them information. RL, of the policy gradient style that works for LLMs, computes a single scalar reward, often binary, denoting success or failure at the end of a long rollout which can often be hundreds of thousands of tokens long. In a naive sense, this means that RL accumulates 1/rollout_length bits per sample compared to pretraining. What is worse is that computing these rollouts requires decoding the actual model which is much more expensive than prefill due to the memory-bandwidth-limited nature of transformer inference at long sequence lengths. This makes RL doubly inefficient – first we waste substantial compute doing rollouts vs prefilling and then for each rollout instead of computing the loss on all tokens we only get a single scalar. Finally, even the reward functions themselves are often crude. People see great results with merely binary reward indicating task failure or success, where a lot of the time the rollout fails for some silly reason such as incorrectly calling a single tool or forgetting to put \boxed around their answer.

The arguments for the profound inefficiency of RL seem compelling. However, empirically, reality works essentially the opposite way. Far from using information inefficiently, RL is vastly more sample efficient than pretraining. RL can achieve extremely rapid gains in performance, often taking a model from being barely able to do a task to completing it near-perfectly, in only a few hundreds to thousands of steps. Moreover, strong gains can be seen with RL in often as few as ten steps. Achieving anything in ten steps with pretraining is basically impossible. However, from the information-theoretic perspective this is exactly backward. What possible information could RL have derived in these ten * num rollout reward scalars that is so fundamental?

Moreover, LLMs are additionally surprising in that they do not use many of the core RL techniques developed in the last decade of independent RL research. From a pure-RL researcher’s perspective, the methods used in LLMs today are very primitive. They are just slight variants of policy gradients and are extremely sensitive about being on-policy, while old-school RL has a complex network of value functions, MCTS-style policy search, replay buffers with intelligent routing and selection of prior memories, model-based rollouts inside the latent space, and so on. How is LLM RL so ridiculously effective when it uses such relatively primitive methods, and secondly why do the methods that we know work well in other RL scenarios not work for LLMs. It is not like nobody has thought of giving an LLM a value function or a replay buffer during RL; these methods just perform poorly and are outperformed by regular GRPO-style policy gradients.

We also know it is not pure on-policyness that is vital to the performance of RL. RL achieves performance gains that regular iterative SFT on on-policy traces cannot accomplish, despite SFT being essentially pretraining-equivalent in the level of information density the loss provides. The standard bits-per-token view above would predict that iterative-SFT on the models’ own successful traces would utterly destroy policy gradient RL. Of course, the opposite occurs and in practice iterative self-SFT is not as effective.

Another interesting phenomenon is that LLM RL is incredibly sensitive to staleness and any numerical differences between trainer and rollouter. Tiny numerical differences that appear to make no difference to output quality, and were routinely ignored in inference stacks previously, suddenly become decisive, causing RL training to destabilize and collapse. Unlike old-school RL, even moderate degrees of off-policyness appear highly damaging and also causes performance plateaus or instabilities. By contrast, old-school RL was basically the wild west. Atari-style agents do not care about on-policyness at all. You can have a replay buffer with traces from whenever and the agent just chugs through them and learns. People did distributed RL back in the day and nobody cared to match their inference vs trainer implementations down to numerical precisions; the training just went on regardless¹.

This is all profoundly mystifying. Somehow all of the intuitions and understanding we have of pretraining, the previous paradigm of RL, and general information-theoretic reasoning all appear completely incorrect when it comes to LLM RL.

So what is going on?

First, I will try to give an explanation for the information-theoretic discrepancy which can explain why RL does work and why it works so efficiently for LLMs. Then, we will propose a broader theory of SNR ratio and the bias-variance tradeoff for LLM training across training stages to try to understand why LLM RL has such different characteristics compared to regular RL.

So, turning to the information-theoretic approach, what can explain RL’s apparent efficiency despite all the theoretical approaches saying it should be extremely inefficient? This puzzled me for a very long time since initially I agreed with the Ord-style reasoning. The continued and ever growing empirical successes of LLM RL were thus pretty mysterious to me. However, recently, I think I have at least a hypothesis that makes it clear how to reconcile these two things.

The standard decomposition given by Ord and Dwarkesh is essentially computing the bits of information per token. Pretraining computes the loss on every token so it gets up to logV bits per token where V is the vocab size. For the sake of argument let’s say that RL computes one binary reward per rollout, thus it gets 1 bit per rollout, which can contain hundreds of thousands of tokens. Thus RL obtains potentially many orders of magnitude less bits per sample than pretraining. Since each token in a rollout or in pretrain costs the same number of flops (technically rollout is worse due to memory-bandwidth-bound decoding), then this means that RL should be many orders of magnitude less efficient than pretraining. So how do we reconcile this?

We know it isn’t the data. Doing SFT on RL rollouts does not magically produce orders of magnitude efficiency gains over policy gradients; in fact, it is worse. Rather it is the objective, since that is the only remaining difference. The objective determines, in some sense, what the bits are being used for. In pretraining, the bits are used to predict every single token. In RL/policy-gradients however, the loss is precisely scoped to succeed at the desired task. Importantly, the RL loss is the actual loss we care about². By contrast, we do not actually care about being able to predict the next token very much. This means that while pretraining does in fact give us a firehose of bits, these bits give us information about the wrong loss. Pretraining only learns about successfully completing the desired task incidentally, while for RL this is the entire loss. This means that RL gives vastly fewer bits, but these bits are precisely targeted.

Another way to think about this is from a signal to noise ratio (SNR) perspective. An RL rollout only gives a single bit of information, but that bit is all signal³. Pretraining gives a huge amount of theoretical bits, but these bits are largely unrelated to the task. Predicting some specific token in the middle of the rollout likely is not vital to the outcome. This means that all these extra bits, and the extra gradients they induce, is effectively noise from the perspective of succeeding at the task. The bits RL uses are there in pretraining, but they are potentially drowned out by all the other bits. The SNR for pretraining on the RL objective is thus actually potentially much lower than for RL, despite the orders-of-magnitude more ‘information’ available. This explains why RL can succeed where iterative SFT tends to fail – by computing a loss only on the overall success or failure, RL actually extracts the relevant signal while in SFT this signal is swamped by the next-token-prediction noise. In RL the only gradients point towards succeeding the task. In SFT, there are gradients for succeeding at the task, but also vastly more gradients for all kinds of incidental behaviours for predicting all of the next tokens.

This also explains why RL is so ridiculously sample efficient when done on LLMs (while old-school RL on Atari etc is the opposite and is incredibly sample inefficient!). During pretraining, what happens is that the model quickly approaches and then asymptotes at some rough SNR equilibrium where SGD cannot decrease the loss anymore. Effectively, the signal the model is getting from the loss becomes lost amid all the other noise so the actual parameters effectively diffuse semi-randomly across the loss landscape. The SNR during pretraining effectively sets the ‘resolution’ of features that the model can represent and the loss landscape is some kind of fractal landscape of valleys-within-valleys whereby the model descends into a valley of some height determined by its SNR but cannot descend further into the sub-valleys within the valley due to the noise so it just effectively bounces around skittering over the entrances to the deeper valleys without ever entering.

We actually see the same effects in pretraining when increasing the SNR in other ways. For instance, decreasing the learning rate and increasing batch size both tend to increase SNR⁴. If we pretrain for a long time at some learning rate, and then suddenly drop the learning rate, we see a sudden and dramatic decrease in loss and improvement in capabilities⁵ as the model suddenly gains the ‘resolution’ to be able to descend into a deeper valley which previously it was skating around the top of. This descent also happens very quickly within hundreds to thousands of steps, just like RL, and then it plateaus again at the lower loss-level implied by the greater SNR.

The interesting difference here though is the nature of the objective. When we anneal the learning rate during pretraining, we rapidly increase the SNR temporarily, but we keep the NTP objective. This means that, in effect, the model learns to dramatically improve on next token prediction, which is essentially a massive ensemble of a huge variety of sub-tasks, most being extremely weird. This causes the model to improve a small amount globally, while not improving any specific task that much. RL, by contrast, focuses all of that signal into a very precise objective of succeeding at some particular kind of task. This leads to a very sudden and dramatic improvement in this task while having little effect (or minor degradation) on all the other tasks pretraining taught the model how to do. Suddenly we go from surfing across the entrance of a broad valley to rapidly plummeting into the depths of a valley for a very specific task and that task only.

This also explains why RL performance is ‘jagged’. The objective itself is jagged. We only care about optimizing reward on some small subset of tasks, and we have high-ish signal for these and zero signal for any other tasks. This is very different from pretraining which even when predicting some generic sentence implicitly trains a large ensemble of slightly different tasks. RL does not magically generalize far, but neither does pretraining. Pretraining only creates the appearance of generalizing further because of the incredible breadth hidden within NTP.

Stepping back a little, although we have shown that RL training dramatically increases the SNR of the gradients for the rewarded task, the general bits-per-sample argument is still correct, it is just that these few bits are all you need to get dramatic improvements when they are focused narrowly. However, this also explains why RL only works upon already extremely strong LLM base models. RL still faces a fundamental paucity of bits. It can successfully solve tasks where the required change is relatively few bits away from the base. It cannot solve tasks, such as learning language from scratch, which require many many many bits. The arguments in e.g., the Dwarkesh post about needing to first reach a nontrivial pass-rate via pretraining is absolutely correct. This is why as we require RL to target harder and harder tasks, ‘mid-training’ has risen to such an important art. Regular pretraining on internet text is great for learning English and general facts about the world but contains too few examples of naturally ‘agentic’ behaviour for these to be learned to any reasonable frequency. Thus, to create a suitable base for RL to begin we need to ‘mid-train’ on data similar to the behaviours we want to hone with RL. In this way, the model can start to learn these behaviours using the firehose of bits that is pretraining. Mid-training then serves two fundamental and intertwined purposes. Firstly, it needs to propel the model into a regime where there is a reasonable pass@k success for RL to begin with. Otherwise, the actual bit-rate of RL training is poor since basically no rollouts ever succeed. Secondly, it needs to minimize the number of bits required to move between the pre-RL base and the final RL model, since RL only has the capacity to move the parameters a short distance⁶, but in a very, very precise direction.

Stepping back even further, this provides a good perspective on the fundamental differences between LLM-style RL and the original RL paradigm. Original RL, such as in Atari and AlphaZero etc were principally focused on RL from scratch. Here, RL had to provide all of the bits necessary to learn from scratch. In this setting, the bit-poverty of RL bites extremely hard and explains many of the challenges of original RL. Firstly, unlike LLM-RL, everybody found that classical RL is incredibly sample inefficient. I.e. it would take millions of hours of gameplay to teach an agent to play even simple Atari games which a human could pick up in minutes. Of course, this is explainable now since, like the LLM, the human has already learnt all of perception and generally acting in the world and just needs a few high signal bits to adjust their policy a tiny distance in parameter space to suddenly adapt. Secondly, it explains why scaling worked for LLMs and not for classical RL⁷. If you are saturated on bit-rate, then adding additional parameters does not meaningfully help. If you are trying to fill a cave with a tiny trickle of water, increasing the size of the cave will buy you nothing. On the other hand, the NTP loss provides an absolute firehose of bits from every sentence, and thus requires vastly larger models to properly saturate that flow. Thirdly, classical RL was also substantially bottlenecked by exploration. This is because there is a direct loop between the agent’s actions and the data it sees, so the agent can effectively get itself stuck in some degenerate loop where it generates bad data and so learns a bad policy and then that bad policy locks in further bad data, and so on. LLM RL conveniently sidesteps this using the breadth of pretraining. Crucially, pretraining is completely off-policy so the pretrained model has already sucked up much of the information in the entire world. This provides an absolutely massive default policy space for the RL trainer to operate on. LLM RL simply does not (yet) have to explore⁸. Instead, if necessary, we patch lack of exploration in RL with simply getting ‘better’ midtraining data. Where this data is designed to cover the desired distribution required for success at the RL task.

More speculatively, we can relate much of this to the bias-variance trade-off. For instance, why does classical RL do so well with off-policy learning and can thus tolerate massive replay buffers while LLM RL is so ridiculously sensitive to staleness and the tiniest numerical differences between trainer and rollouter? Off-policy methods reduce variance at the cost of introducing bias, since the model gets rewards from policies that are not its own⁹. When the bit-rate is tiny, as in classical RL, variance is the biggest problem since it stops us from learning from the few bits that we have. Bias is much less of a problem since the bias and the true objective are often highly correlated when little training has taken place¹⁰. In fact, bias can be beneficial since it can artificially restrict the possible space, which makes this restricted space easier to learn. However, during LLM RL, the pretrained base has already undergone an absolutely vast amount of training using the unbiased NTP objective. It can resolve features extremely far down the scaling curve, and its parameters are highly tuned, and thus sensitive, to the precise specifics of the true distribution. In this case, introducing bias suddenly is catastrophic. It yanks the model from its extremely carefully constructed position in some sub-sub valley and just pulls it in some other direction where, of course, when this level of amortized compute has already been compressed into the model, almost all other directions are bad. This can then explain why the LLM models are so sensitive to bias when classical RL is not – LLMs are so much further down the scaling curve that really no bias is manageable and the entire game simply becomes around counteracting and minimizing variance to gain the resolution to inch a step further down the fractal spectrum of features.

The bitter lesson, then, can also be reframed in these terms. The bitter lesson essentially says that variance can be handled with more compute or data, while bias cannot. Thus, methods that scale are those which are as unbiased as possible even if they have high variance. At small scale, bias is good and necessary since the variance of the unbiased methods is too high and without certain simplifications which induce bias your model has no hope of grappling with the true complexity of reality. At larger scales, however, we can use our compute and data scale to suppress the variance which made these simple methods work poorly at small scale, and also the benefit of bias diminishes and eventually becomes harmful as the raw capacity of our models increase to match the inherent complexity of the task¹¹. This means that in the long run, the principal game is simply continually using scale, data, and algorithmic tricks to indefinitely reduce variance and increase SNR of a core unbiased algorithm. As we can keep increasing SNR, we can resolve ever finer features of the loss landscape and improve performance, but obviously as we penetrate deeper down the curve, each feature is rarer and more provisional, and thus requires an ever higher SNR to reliably capture it. The game then simply becomes how far we can push SNR and whether the marginal gains of doing so are worthwhile.

At least this is broadly how I am thinking about these things now.