RL-Free Weak-to-Strong Natural Language Autoencoders

Gao & Shulman & J.Hilton show that compared to RL, best-of-N SFT spends much less KL distance for the same amount of optimization.

In Redwood's Consistency Lens, which inspired the idea of Natural Language Autoencoders, it is claimed that the activations of neural networks correspond close to the structure of text, thus it is plausibly possible to find a "vec2sentence" (akin to word2vec) mapping, that needs not have a lot of intelligence in it.

Because GRPO generalizes, and SFT memorizes (e.g. memorizing a mapping, a vocabulary list), it is a superior choice when wanting to avoid steganography. Initial experiments confirm that best-of-N SFT spends an order of magnitude less KL budget (measured via drift proxies) while recovering 78% of the GRPO reward. However, only with 1% of Anthropic's compute budget on the Qwen-2.5-7B RL run.

Scaling up the experiment is dead simple; one just needs to set the advantages to the argmax to obtain best-of-N SFT from GRPO.

Furthermore, once the structure of a weak model is understood, one may posit a platonic representation of LLM thought, meaning we can warm-start a stronger models with the thoughts from a weaker one, recursively. NLA-generated thoughts can already be transplanted in this way, with acceptable loss. (cf. cross-model transfer, LW). The corresponding experiment is again very simple, train Qwen2.5-7B down and implant into Gemma-3-12B.

Regarding secondary endpoints besides loss, weak-to-strong and SFT have the further advantage of keeping the activation verbalizer stupid, meaning it is not able to reconfabulate what the LLM is thinking from shallow activation vectors that just represent the problem statement, in other words, a sufficiently "advanced" thought is strong evidence of it being actually read out. This can be operationalized with math benchmarks that we know the activation verbalizer cannot solve. (Oakhu & Greenblatt, Alignment Forum)

Longer Draft, LW