AI researchers debate how close we are to recursive self-improvement

I'm chatting with three AI researcher friends from whom I learn a lot every time we talk. They also happen to be at somewhat open-ish labs and companies, so they can actually say things on the record. I'm joined by Beren Millidge, CTO of Zyphra, which is developing open source models; John Schulman, chief scientist at Thinking Machines, previously a co-founder of OpenAI who led the RLHF work that led to ChatGPT; and Charlie O'Neill, head of model training at Baseten.

The first question: if we're in 2036 and we don't have billions of crazy superintelligences running around that have radically transformed the world, what is the most likely reason that doesn't end up being the case — other than exogenous political shocks, a war, or a ban on AI? What is the most likely technical reason that 2036 isn't a crazy alien superintelligence world?

Why 2036 Might Not Be an Alien Superintelligence World

There's a classic pattern, almost like Moravec's paradox: we think of AI as "if it can do this hard thing, it will be amazing" — solve hard maths problems, win at chess, and so on. Then it solves those things, and the impact is real but not transformative. If that pattern continues and the true spark of generalization never occurs, AI could end up extremely good at everything people put into a benchmark or an environment, while some persistent sim-to-real gap blocks everything else. I think this is unlikely — we already see this kind of generalization from RL in practice. But if meta-learning generalization is ridiculously hard, and continual learning proves impossible to solve, this would be my default scenario in that case.

I agree. Humans have a lot of advantages over current models. Each new model catches up in some areas, but you end up bottlenecked by the places where the model is weaker — where it has worse judgment, or can't check itself well enough. There's a repeating cycle: a new model comes out, people are blown away and declare "this is it, this is AGI," then after a month or so of use it starts to feel dumb. That cycle might just keep going, and it's hard to predict how many times it repeats. Right now you don't get explosive growth in capabilities because you're still bottlenecked enough when doing research and engineering. Even if a model can write far more code than a person, it doesn't make you 100X more productive. So maybe there are just more of these cycles than we'd expect.

How far is the current recipe from the global optimum?

For me, the question is how far off the global optimum of "a learner you could have on a chip" is from transformer + RL — basically the current recipe. People imagine that once you have an agent even 0.1% better than all humans at AI research, the ability to run hundreds of thousands or millions of copies in parallel, faster as chips speed up, will outweigh every other bottleneck and produce a very fast takeoff in self-improvement. I could imagine that if we continue on the current trajectory — self-attention, RL, scaling up RL environments.

Think about Moore's law: a nice straight line that held for a very long time, but it took many discrete discontinuities and innovations to keep that scaling going. The same has happened with LLMs. We had the pre-training scaling law, it hit diminishing returns, then RL came along and solved that, producing a new curve of diminishing returns that kept the line looking straight. If solving the next bottleneck requires another such discontinuity, I'm not sure that training LLMs with RL environments — even RSI-targeted ones — would be able to discover it. If not, we'll probably hit an asymptotic curve.

But do you think that discontinuity would be harder than anything since 2012? If we had the answer, we'd basically be able to implement it. But we should distinguish between a discontinuity that adds to the current paradigm cumulatively — something beyond RL that remains to be discovered, which the models might be capable of connecting the dots toward — versus how far off the global optimum we are: whether we'd have to throw out gradient descent and neural nets entirely. I don't think an LLM, no matter how many you run, is necessarily capable of discovering that if it's too far away. The only hope is if deep learning can't get us to an AI that can at least dominate human research and development, including the human ability to come up with new paradigms. Or maybe humans would also never have discovered the next learning architecture. But to the extent humans could have discovered it eventually — if you just extrapolate the progress from 2012 to now, even though it's been powered by huge compute scaling, it would be weird if it didn't reach the point of dominating humans, at least in R&D, over the next few years.

The chess Elo discontinuity

Ryan Greenblatt made this point recently on the podcast: as AIs get more capable, they could make progress on simulations that incentivize getting better not only at AI R&D but at science generally — something all the labs and many startups are targeting. As an intuition pump, look at the Elo scores of chess bots since the '80s: a very linear increase over time, but a huge discontinuity as they cross the human range — from human experts always beating AIs to never beating them — even as the Elo increase stays linear. I agree that so far AI capabilities haven't been a big deal in end economic impact, but that's because they're still slowly rising in Elo relative to humans.

I agree it would be very surprising for that not to happen. The only way it doesn't is if, as you said, capabilities asymptote just before — because we're already pretty close, in my opinion, to crossing the human Elo score. That's the only way, in the scenario where we're sitting here in 2035 and everything is normal. The only other way is dramatic regulation on AI.

This is what I see as the most likely way for this scenario to happen, actually, rather than a technical thing. There are different kinds of research. There's research in the autoresearch style, where the objective is already specified very cleanly and you're optimizing that objective. Everyone is picturing that if we continue along this path of making pre-training loss go down and making our environments' rewards go up, that's going to lead to improvement. But maybe what Ryan is talking about is the much more open-ended type of science required for paradigm shifts, where we can't specify the objective — and the AIs definitely can't specify it either. We have to be really careful about how we specify objectives for any of these things.

Maybe your point is that the nature of the breakthroughs since 2012 is that we found something we weren't optimizing for at the time. Presumably a big breakthrough was realizing that next-token prediction was the thing to do. You wouldn't have thought the nanoGPT speedrun was the thing to optimize for in 2014. But now that we're in this new paradigm, you would think to run a speedrun on it and have AIs get really good at it. Maybe there's a next inner loop to optimize that the AIs wouldn't anticipate. There's an outer loop of revenue or something that eventually should be strong, but it's a very slow outer loop.

I remember from the early OpenAI days having the intuition that just minimizing log loss wasn't going to get you to intelligence, because the important bits account for such a small fraction of the loss that they'd be overwhelmed by noise. So just training a language model on next-token prediction wasn't going to learn the interesting things you want it to learn; we needed to craft better objectives that put more emphasis on the important things. You can make all sorts of arguments for this — for example, humans probably don't learn to model everything in our environment; most people can't create a photorealistic reproduction of a scene they've looked at, so we must need a better objective. But then it turned out that it just worked anyway. As you were pointing out, the inner loop even in current AI research — post-training benchmarks or whatever — doesn't necessarily translate into what users like.

The whole field relies a lot on generalization, and it's very hard to predict when you're going to get generalization, or out-of-distribution generalization. We know that if you train on the task you care about, you'll do better. But the most important advances are often types of generalization we have no right to expect. For example, from just pre-training on the naive next-token-prediction objective to various tasks that require a deep understanding of the input, or learning a skill from pre-training that's very rare and not heavily represented. Then there's also generalization from verifiable tasks to less verifiable ones — again, no reason a priori to expect it.

This is an interesting question, because one intuition pump for why you'd see a very rapid singularity — without even scaling up inputs to AI progress beyond AI labor — is that before every seven-figure experiment you run, you spend an equivalent amount of compute on AI labor. You'd have automated versions of you spending a century thinking about what the optimal experiment is, doing small-scale ablations, developing literally a century's worth of theory going back even before deep learning. Before deciding what experiment to run, you'd set it up extremely optimally, and then do a century of thinking afterward, analyzing what happened and what the next experiment should be. If you think hard enough, you probably could have expected some of these things beforehand. There's probably some very clever small-scale experiment that lets you build theory that then generalizes to the large-scale experiment. So I'd expect we're nowhere near the ceiling of how well you can do research. I imagine a future where AI is doing a lot of analysis and theory building, spending compute comparable to what you spend on the experiments themselves, building theory around what we've seen so far.

There are really concrete examples of this when the objective is well specified. All thinking can do is update your posterior based on the bits you've gotten since you formed your prior — you can't gain new bits from just thinking. But when the objective is well specified and the data is sitting around, I imagine there will be a big speed-up in the current paradigm. A good example: if you had an AI think about the Kaplan scaling laws, an AI today would have noticed that they just took intermediate checkpoints and didn't account for annealing, so the result was wrong. That would have been caught years earlier, cutting off a year or two of wasted progress. Once the objective is well specified — lower pre-training loss or whatever — there are many examples where just thinking a bit more would have significantly cut down on work that was done.

Things like muP, how learning rate scales with model size, and realizing that model width matters too — I feel like you can back out a lot of these things and cut off a lot of low-hanging fruit. I'd imagine a 10x speed-up if the AI's objective is just "maximize the objective we're currently on." But I don't see how that generalizes to coming up with the right objective in the first place. Just thinking doesn't necessarily buy you the right objective.

I think this is really the key question for any kind of very rapid recursive self-improvement from current AIs: how well can AIs generalize to learning their own objectives? To have any self-propelling automated loop, the AI needs to propose objectives, optimize them, figure things out, propose a new objective, and have this not go off the rails at any point for a long, long time.

Coming back to Moravec's paradox, there might be a case of it here: we think this kind of autonomy — being self-encapsulated enough to think about what we should do, then go do it, and keep that loop running — is super easy because we always do it. Evolution, after all, had to create creatures that survive by themselves for long periods. This might just be something that is, for some reason, really hard for AI, the same way locomotion is really hard while math is super easy despite being super hard for us.

But doesn't the increasing time horizon suggest that's changing?

Yeah, exactly — that's another possibility, though I agree there's no obvious evidence for it. In fact, the fact that our agents are now super persistent and it's quite easy to run these loops is kind of evidence against it. But if this autonomy is hard, it could be one reason we don't get an immediate takeoff.

Looking back from 2012 — or from when you started your research — until now, which of all the innovations, purely engineering or purely conceptual, seems like the thing humans would have had to do last before AI totally automates AI R&D?

Probably just iteratively asking the right questions. Even if you can get the AI to run any experiment, you still need to decide what experiments to do. Right now, AIs aren't very good at this compared to coding the experiment. Whenever we talk about research with them, they propose a bunch of miscellaneous, very tiny steps. Consider the jump from DeepMind's approach — "we'll solve intelligence by learning to play games at a superhuman level" — to one random researcher, Radford, saying "I'm going to try just predicting the next token over a very wide swath of data." Even after Radford discovered that, it took a while before people decided to scale it up, because we had to come up with scaling laws and the insight that you could very reliably predict these things.

I'd say the last job for humans — the role that will last the longest — is defining the objective and deciding what we actually want. Deciding how AI assistants should behave, what it means to be helpful, or what the objective is in RL from human feedback is one such thing; later, defining constitutions and model specs is another. Even if AIs can do all the technical work, we'll still have to do a lot of that. Alignment is the final job.

Alignment is sort of the answer. But alignment itself decomposes into specifying the objective — figuring out what the right objective should be — and then actually achieving or optimizing the objective you've defined. I think the first part isn't going away anytime soon. If you think about a post-training team and why it needs a lot of people, it's because there are many different areas where you have to figure out how the model should behave. It would be very hard to automate the whole thing, because someone has to think about how the model should behave in each area.

The thing that I think is most impressive about Antithesis is that we started using it on a team that was building high-assurance software and being really careful. Nonetheless, it was able to shake out bugs that were otherwise going to be really hard to find. That's important both because it helps make those systems more reliable and because it helps the teams that build them move faster. This matters more and more as code production is increasingly automated.

In general, as we've been using agents more and more, the key problem you run into is the verification bottleneck: the time it takes for people to look at code and figure out whether it's actually something you want to accept into your production software. Tools that make testing better are incredibly helpful there — they ease the verification bottleneck and let you get more done and move faster, because you can have more confidence that agent-generated code isn't introducing new problems. To see how Antithesis fits into your development process, go to antithesis.com/dwarkesh.

Why hasn't there been massive consolidation among model providers, given the strong forces pointing toward centralization?

Distillation is the main counterweight fighting centralization. Anything learned through reinforcement learning (RL) represents a relatively small number of bits that can be learned from a modest amount of data. If you can capture trajectories exhibiting a specific behavior from a frontier model, you can easily distill it.

Another factor is the emergence of company-specific models that learn continuously from deployment. However, continual learning does not prevent distillation; even if a frontier model improves daily, competitors can distill it at the same pace.

The Role of Prompt Distributions and Router Data

To distill behavior effectively, don't you need to know the right prompt distribution?

Yes. For supervised distillation, prompt distribution is critical. Even with full access to chain-of-thought outputs, distilling all useful capabilities is difficult without a wide distribution of realistic prompts.

Chinese labs benefit significantly from router and proxy services. These services allow users in China to access blocked US frontier models, primarily for coding tasks, while collecting and selling user trace data. This provides distillers with an ideal prompt distribution.

Modern pipelines combine human-provided seeds with this real-world traffic, using existing frontier models to synthesize vast variations and automate environment creation. As models improve, humans need to provide fewer bits of information.

Distillation vs. Frontier Training

Doesn't capturing multi-step user workflows still bottleneck distillers to services with actual user traffic?

Not necessarily, though traffic is very helpful. Distilling is often easier than training a frontier model from scratch. To create a domain-specific capability—like an AI politician—a distiller can simply prompt a frontier model that already possesses general capability to generate traces, and then synthesize a billion variations. The original frontier lab, by contrast, had to collect primary ground-truth data and construct the training environments de novo.

Implications for Frontier Labs

Access to realistic user prompt distributions explains why some open or fast-following models rival or outperform frontier models despite lacking native access to proprietary RL environments. This suggests that frontier labs may hold less of an advantage in synthetic RL environments than assumed:

• Real-world deployment data and realistic user distributions may matter more than isolated synthetic training environments.
• Alternatively, frontier labs attempting logit distillation on their own smaller models may be hitting a student-teacher capacity gap, limiting how effectively those behaviors transfer downward.

The difference between Opus 4.6 and Opus 5 is that Opus 5 really feels like it's got this AI-as-a-judge checking every possible thing it's done. That's why it uses so many tokens. It tries to think about all these things, but it doesn't necessarily have the big model smell of Fable to know when to stop doing that, or when's a good path to go down. The reach exceeds the grasp.

I would offer a slightly different hypothesis. There are a couple of different axes for the environments you can create: one is difficulty and the other is realism. It's comparatively easy to create a lot of difficult environments that involve doing a much more complicated task or something that requires a lot more cleverness. You could call this the benchmaxxing distribution, because a lot of the most prominent benchmarks just involve doing some very hard puzzle-like task that's easy to verify. Then there's the realism axis, where you want the model to be good in the realistic coding agent setting, with multiple back-and-forths with the human and multiple objectives. The labs crafting the model behavior for the first time need to push in both directions. To get good model behavior, you need to really push on the realism axis and have rubrics or some kind of human feedback informing the reward function you use there. But if you try to do distillation naively, you end up just matching the teacher on the benchmaxxing distribution. If you don't have enough of the environments that really exercise the capabilities in these trickier realistic settings, you're not going to get those into your student model.

Maybe one thing that's happening is that the big models generalize better from the tricky narrow tasks to these more realistic tasks. If you have a really good realistic prompt distribution for distillation, you can match the big model really well. But if you only have this distribution of easily verifiable tasks, you can match the big model on all the benchmarks yet do worse on this broader distribution. That might even explain something about the smaller Anthropic models, like Sonnet 5, though it's hard to predict exactly what they're doing to post-train those models. It could also be that they're always changing their post-training stack and just got a few things wrong in some of these models—I don't know, maybe they turned something up too high and created some quirks that people really don't like. It's really easy to screw up post-training in some way that doesn't show up in benchmarks.

One other very basic point: the frontier AI labs buy all their data from big data companies. The Chinese can also just buy the same data from data companies—and they are.

And they are. Exactly. There are a lot of people being annoyed about this, but if they have exactly the same data and can buy it, they can also distill. It means it's quite easy to keep up, really.

The open question is how the first models capable of automating AI R&D will actually be trained. There's a toy version — the thing Ryan was describing — where you have GPT-8 try to build GPT-3-sized models that are really good at inner-loop type challenges: beating video games that require continual learning, or reaching a certain loss with the least amount of compute. But John made an interesting point that this may not be how it happens in practice.

In practice, we'll probably do some combination of learning from human feedback to absorb the researchers' taste, and creating a lot of practice environments involving multi-step research projects. Each iteration, people will patch whatever seems most broken in the last one: researchers will be using the AIs heavily, notice consistent weaknesses, and fix those either by collecting human feedback or by creating environments.

A useful way to think about it is how much of the lineage you roll back and then let self-play run from there. In the limit, you'd just give the model a GPU and maybe some neural nets and say, "Figure out how to train a model to do these particular tasks." The way it currently works, you stay at the very edge of the lineage: take the bugs Anthropic has found in its training stack over the last few months and turn those into environments. You need to train and get better on the frontier, so you lock in all the previous history. But you could imagine rolling back to before GRPO, say, with environments that get the model to discover the best form of RL on its own, then rolling further and further back. Still, we'll likely remain so compute-bottlenecked that people just stay at the frontier, diffing the bugs and improvements found since the last model version and turning those into training environments — which is also great for having non-stale, new data between model generations.

This is basically continual learning within the AI lab: distilling the last three months of AI research progress, through environments and RLHF-type methods, back into the model itself. That's maybe why some of us feel it's asymptotic — you're always just trying to capture the last three months of progress, which is partly contributed by AIs but still has humans in the loop. It feels like constantly inching closer to what human researchers are finding and capable of. One caveat: if you're only distilling on trajectories, you can never exceed them. But environments can go far above what a human can do — it's easy to design an environment no human can solve, while the AI can still try. That would be the path to getting ahead of human-driven AI research.

Verifiable goals as training sets

An example of such a training set for recursive self-improvement: nanochat speedrun, but done even faster than a human speedrunner. In AI research especially, it's easy to define goals — say the loss needs to be 1.3, which no human can achieve now, but it's an extremely measurable, verifiable task. Or building a 100-million-parameter model that beats Minecraft — maybe too easy — or beats a much more complicated game.

It is striking that 100-million-parameter models beating Minecraft is now "too easy" — imagine saying that five years ago.

Most research isn't hill climbing

A lot of research isn't like that, though — hill climbing on a well-defined goal. It's more like: here's an intuition about some way models should be better, and an algorithm idea that seems to move in that direction. So you design a task meant to show signs of life on that approach, and check whether you get them. If you do, you make successively more realistic versions of the task. It's guided much more by intuition; the inner loop tests the intuition rather than the test itself leading to the insight.

Right — you're not directly optimizing the eventual objective you care about or the practical production objective. You relax the objective a little, on the realism axis, find methods that actually work, and then try to get back to realism once the method matures.

There's also research oriented toward explaining things and developing theory. We often don't have mathematical theories in machine learning that are very predictive, but we do have a lot of informal theories for what's going on.

Will the loop close without humans?

Presumably the models will be trained on some combination of all of these tasks: some easily verifiable, some LLM-as-a-judge, some just asking a human, "Does this look reasonable?" The hope is that these generalize to much harder, vaguer, fuzzier tasks. It probably will to some extent — but whether it generalizes enough that the loop can become self-sustaining without humans in the loop at all is unclear.

Taking a step back, here's how I'd characterize the plan for AI research going forward — tell me if you agree. The bet is that we will scale up RLVR training across millions of diverse environments and hundreds of different domains. What emerges at the other end is an agent that has learned basic — or less than basic — skills: persistence, triaging information and context, and eventually end-to-end optimization of working with other agents. Such an agent will be very sample-efficient within context — you've done research on scaling in-context learning to make it arbitrarily long — and what comes out will function like a drop-in remote worker over the course of a week or a month. First, do you agree that this is the bet the labs are making? And second, is it enough? The idea is learning how to learn within these simulacra in a data center, then getting deployed into the real world without actually learning from real-world deployment — only acquiring these meta-skills from simulated environments.

It's now hard to separate how much of the labs' effort goes toward direct RSI versus making generally intelligent models they can keep deploying to collect revenue and fund the next big training run. For the latter, yes, that's probably just the bet they're making.

The pattern of where these environments are going is very clear, and Anthropic's lineage of environments is a good example. First, focus on coding and get really good at that. Coding is probably the lowest-hanging fruit in terms of internet data available to create environments, plus their own internal material they can turn into environments. Then generalize: go up to finance next — there's so much Excel data and similar material for RL training — then PowerPoints, this long tail of the working economy. That seemed to work really well, and a lot of other labs, even open source ones, have now realized it was the correct bet to make.

But what's the implication? When I had Dario on the podcast, I asked him: if you truly expect models that are human-like in their ability to learn on the job, why bake in skills like working with PowerPoint? Wouldn't you just expect the model to pick that up while deployed? There are multiple explanations. One is that we expect models to get there soon but they're not there yet, so why not amortize these skills into training. Another is that we're not focused on making it great at widely deployed work — we just want it great at RSI, and this is a way to get revenue to pour back into a model that's actually good at RSI development. Then once the singularity happens, what comes out the other end will be good at all the things that seem like bottlenecks to the current generation.

John, do you have takes on why there's so much task-specific knowledge in these models if the path is this kind of generalization? If the models were good enough at learning in context, then in theory you wouldn't need to train them on finance — they could just read all the books on the fly and figure out how to do everything in the appropriate jurisdiction.

You could argue that domain-specific training is needed just to make them more efficient. Even if they were smart enough to figure things out on the fly, you might still want to do a bunch of RL and bake those intuitions into the weights so the model is more efficient at runtime. In practice, model providers do seem to be going domain by domain, strengthening the models in the highest-value domains. I'd say that's one reason models have gotten so much better — the providers have covered a lot of the high-value domains and the most common types of skills.

Another point is that it's not that expensive to do both at the same time. The models are massive; they can easily afford, in terms of parameters, to learn everything. There's likely some transfer: even if finance isn't specific to RSI, the general meta-learning — how to figure out what's important, how to have taste, how to do long-horizon work — is potentially generalizable. There's also not that much RSI data in the world; it's hard to generate and requires a lot of effort. So if you can amortize this other data in, you get some transfer from it. You already have masses of compute and parameter space, so why not do that as well — on top of, obviously, the direct commercial intent of selling a model.

I'll add one question: whether this current sim-to-real paradigm will be the dominant one forever. You look at what real-world tasks are like, then try to create environments that can be simulated in the data center and run RL on them. This has obviously been very successful, but it has weaknesses, because a lot of things are just hard to simulate — especially anything involving interacting with many humans in real time. So there's some question about whether sim-to-real will remain the dominant framework.

I think sim-to-real has to be the dominant framework while sample efficiency is low, because right now you need thousands and thousands of interactions with humans, and no human is going to sit in the loop of RL training. So we have to simulate that to get the samples you need.

But obviously, if sample efficiency improves a lot, you'd expect learning from deployment to become a much bigger part of it. There are also other things you could do: you can learn off-policy, taking all the traces and potentially learning something from them even without resimulating everything.

Jane Street just launched a new competition, its most ambitious one yet: design a protocol-emulator ASIC. If you have a chip you want to test, you connect it to this ASIC, and the ASIC simulates realistic traffic, so you can see how the chip responds without plugging it into a live system. Jane Street is looking for flexible, general-purpose designs, not single-protocol emulators. When I was chatting with them, they suggested starting with three apparently very common protocols: UART, SPI, and I²C. They also hope more ambitious designs will tackle low-speed USB, Ethernet, and other protocols that flex your chip's specific architecture. Importantly, your design should be reprogrammable rather than smashing a bunch of specific protocols onto a chip—if a new protocol comes out after your ASIC is taped out, your chip still needs to handle it. How exactly it does that is up to you. There is one hard constraint: your design must target an open-source 130-nanometer process node, because Jane Street will pay to tape out the most novel submissions and send physical copies to the winners. The competition is open until January 18, 2027, and working in teams is highly encouraged. Go to janestreet.com/dwarkesh to download the template code and get started.

I want to ask more about this, because it's weird that 50% of compute is spent on inference that isn't directly helping the model become better. One of the key advantages you'd expect digital minds to eventually have is that, unlike a human who gets 50 years of real-world experience, a model will experience, through all its instances, millions of years of deployment across all kinds of economically relevant work. Right now, that data isn't, in a meaningful sense, helping the model get better. It seems so obvious that eventually models should be able to learn from this data. Once they do, you'd have something that almost feels like a widely deployed intelligence explosion, because the model is assimilating so much information across all these deployed instances. When do you expect this kind of hive-mind thing to start happening?

I think broadly, at a very basic level, this is already happening—just in the next generation of models. Right now, you can take your deployment data and put it in the pre-train or mid-train of future models, especially if you do some kind of filtering, judgment, annotation, or synthesization of it.

How much do you think that explains the generation-over-generation improvement?

I think it explains quite a bit. I don't know whether the labs do this, because theoretically they claim not to train on people's data. But the Chinese 100% do—they definitely get this advantage. This is basically what distillation is: they take the models, get some fraction of their deployment data by pinging the model, and then train their next generation on it. They can certainly do it on their own models as well; there's no reason not to whatsoever.

I completely agree. If you zoom out far enough, this is definitely happening. What we're all picturing—the holy grail of continual learning—is this very organic, live loop of an individual model getting an experience and live-updating on the spot and learning from it. A lot of things break when you zoom into that level of granularity. But the big labs are doing this, and there are also early signs of life of people using open-source models doing it at a much faster cadence. A good example is probably Composer; Harvey's doing the same thing with legal agents. You have some sort of model, and you're getting very specific environments from the data you have for that particular task, from the things users are complaining about, and from all the feedback you're somehow extracting from your specific deployments. A lot of these companies have an advantage over the big labs in that they can use this data really, really well. Then they create environments, do a big post-train of Kimi K3, and go deploy it. They might do some online learning as well, like Composer did—basically REINFORCE for a long time. There's still a human in the loop, still a human saying, "Okay, these are the signals we care about. Here's how we're going to create environments from the data that we have." It's still a longer cadence than the one you're thinking of, but it really is happening, and eventually that loop will become faster and faster.

The Composer thing is interesting because this is where, in Cursor, people press Tab or don't press Tab on the next completion the model suggests, and based on that, every single day, Composer gets better at predicting the next—

That was the old Tab model. They actually did the same thing not just for the Tab model, but for the actual generative model.

Oh, I see. Interesting. It's hard because when you do online reinforcement learning, you don't have groups—you just have one user saying one thing and then one rollout, so you have a big variance-reduction problem. Cursor's fuzzy answer to this was, "We have very good heuristics which are able to estimate how much better or worse than average this response was." Then they would do this big REINFORCE update. Their solution to whether it got worse or not was that if it improved on CursorBench, they would deploy the new model every five hours; if it didn't, they would throw that version out.

I think your biggest problem is actually just not knowing what the reward function should be for natural data. If you use some kind of superficial signal, like whether they accepted the edit, that might get reward-hacked in some way.

The question is whether longer-horizon tasks get harder to simulate in a data center. Even in coding, we may already be at the point where no year-long coding task avoids eventually talking to a client, interacting with the company, or dealing with users. If you think about the full range of what we want AI to be capable of, superintelligence should eventually be able to run a business, start a new business and make it profitable, trade profitably in the markets, or win a court case — all things that are very hard to simulate in a data center, where interacting with the real world is an inherent part of the learning.

Maybe models learn these skills through sim-to-real transfer. But alternatively, maybe they need weight updates from those real interactions to improve. If transfer isn't strong enough and weight updates are required, then the models' sample inefficiency becomes a deeper problem. The reason this matters: by default, I don't see how you avoid some kind of crazy recursive self-improvement within the next 10 years. The one reason that might not happen is sample efficiency of weight updates — models seem plausibly a millionfold behind humans, comparing the data a human sees from birth to adulthood against what a model sees from cold start to finishing training. So the two questions are: will there be good transfer from simulations to the extremely long-horizon, complicated real-world tasks we want AI to do? And if not, does the lack of sample efficiency come to bite us?

Cumulative versus non-stationary tasks

A useful breakdown of the tasks models get good at versus those where they keep struggling is whether the task is cumulative, or whether it has a non-stationary distribution requiring continual learning and relitigating. RSI is an example of a cumulative task. It's theoretically possible to have a Python file under a million tokens which, from scratch, trains a model capable of recursive self-improvement. Every discovery is a line in the sand that you hold: if RSI doesn't require discovering a new attention variant, then once you've discovered attention, mixture of experts, and GRPO, you just add them to the training stack and they stay there. A good example is the 5.6 Sol or 5.6 Terra training OpenAI told us about — it wouldn't have to go back and discover attention; it would basically call scripts like pre-training.sh and post-training.sh and run them.

The real world, though — the reason people think so much about continual learning — is not a cumulative task. Imagine an agent acting as a legal associate at a law firm. That's a very non-stationary distribution: it has to fit in its context the relationships between all the important people at the company, which change constantly, plus all the implicit ways things are done and where to find information. That's not as clean a cumulative task as RSI.

There will likely be this breakdown between tasks. But if the labs realize it — and they do believe RSI is cumulative, in the sense that no brand-new architecture needs to be discovered — then maybe more and more effort and compute gets focused on RSI versus the other tasks. It's so unfortunate that RSI happened to be easier than being a paralegal.

Sources of model weakness

Today's models are weaker than humans in many different ways. Some may relate to sample efficiency in a certain regime. In some regimes models are very sample-efficient, like learning in context, but there may be a medium-length regime where they're less efficient, because humans can do some kind of weight update more efficiently. That could be one source of weakness, but there are others that are completely different: lower diversity of thought than humans, or being bad at certain kinds of long-horizon judgments. A lot of what people call taste is behavior that works in the long run, which people have realized works in the long run — not all of taste, but some aspect of it. Especially in software engineering, much of taste is "which systems will be maintainable and work well over the long run of this project?" So there are a variety of weaknesses limiting RSI along with other things; some relate to sample efficiency, some don't.

An interesting thought experiment: suppose you gave a model a trillion-token context window — whatever it would take to fit all your experience prior to, say, RLHF — with the same sample efficiency and in-context learning ability it has at a million tokens. Is taste then solved? Would it make the same judgments you do, or is something fundamentally missing beyond a longer context window with the same sample efficiency?

It would have to be trained to learn from that context — either trained to learn the right update to make from it, or to generalize.

"So you don't think you can just dump it all in, your whole life, your research experience?"

"You still need the data to train it on long context."

Even if you could theoretically get a trillion-token context, you would need a trillion lengths of data to train it. Right now you have 10k context; you can't just dump in a million.

— I'm just asking if you had that. In theory, I think, yes?

In theory, I think yes. It really comes down to the question of how meta-learnable taste is from shorter-horizon episodes. There's no obvious reason it requires super-long context, because humans somehow developed taste without many long episodes — we don't live to be 10,000, and we develop pretty quickly. Consider a PhD: the difference between a first-year student and a final-year student or postdoc is maybe five years, over which they've done perhaps 10–30 research projects in total. Yet they develop taste quite quickly from a relatively short succession of small things. Theoretically, it's possible to develop it that way, and an AI will obviously have vastly more experience in which to meta-learn taste. The open question is how well that generalizes to really long-horizon things, which I think is really unsolved at this point. We don't know.

Going back to the earlier question: eventually there should be a regime where AIs learn a ton from each individual instance of deployment. Currently there's a fuzzy meta process by which models do improve from deployment, but it's a very weak feedback loop. Do you see this hive-mind kind of rapid learning on the horizon, and if so, how exactly does it happen?

Whether we get a hive mind that learns from all of its deployment experience is largely a question of incentives, not a technical one. Companies won't want the model provider to learn from all of their deployment, because that could erode their business advantage. I think the economics will pressure — not necessarily weight updates into one big shared model — but modules that get subbed in. An obvious example is a LoRA, but it might be something else. There's been a lot of work trying to fit arbitrary context length into a fixed size — the linear attention line of work — and cartridges, which are essentially KV caches trained to be very compressed so they hold a lot of information. Those are examples of things companies may be willing to sign up for, if they get subbed into the model without changing the underlying base model itself. There are many versions of learning from your data in real time; the module-based ones don't really help the big labs directly, but I think economic pressure will force the labs down that path first.

— Which economic pressure, though? Even with a bunch of cartridges or LoRAs, you can still take all those traces and dump them into the pre-training of your next generation of models.

Yes. It may be a more indirect form of learning for the big labs, and that's still really valuable to them. But I can't imagine starting off with "we'll directly train one big model on all the exact data we're getting." It will definitely go through stages, because the alternative assumes one discontinuous event where we suddenly fix continuous weight updates. In practice, cartridges and the like let you specialize in deployments; then you generate traces, put them into your model, and three months later release a model that's better at this stuff. You specialize it again, consolidate it again, and eventually the leap gets faster and faster — instead of releasing a model every three months, it's every week, then every day, then every hour, at which point we've basically solved it.

This connects to the question of how far the current paradigm is from being able to do this. There's been a lot of research, including some of our own: at a really large scale, when you wash out enough noise with large enough batches, the outer-loop process of putting data into mid-training and creating your own environments does work in some sort of continual learning regime. But at the micro level — one model, updating it continuously for a law firm or something with a relatively small amount of data — all the methods break down somewhat. If I SFT the model on just successful traces, off-policy or on-policy, then in the very iterative regime, after hundreds of micro-updates, you see catastrophic forgetting: loss of information learned much earlier on top of the base model, and degradation of general capabilities. On-policy distillation pushes this horizon out a bit but eventually succumbs to the same thing. RL is good at getting capabilities in, but not as good at getting knowledge in — explicit knowledge like "this person does this at this law firm, this is a very specific process." You have to pour in a lot of compute to create the right environments to get knowledge in with RL.

— Do you think the fundamental issue — the forgetting and degradation of other skills — is fundamentally one of capacity or of techniques?

A little bit of both. SFT and even on-policy distillation can be way too destructive. The reason RL is so nice is that it changes a very, very small amount about the model — there's a lot of evidence for this; it just tweaks the model within a very small loss valley to get it to the right point. But that also limits what you can do with RL, how much you can actually change the model.

So the reason this isn't winner-take-all, potentially, is that it's just very hard to distill that much information into the base model without ruining something, in an iterative fashion?

It's easy to distill it into a different base model. I think it's mostly technique—it's definitely not that there isn't capacity. If you take a model with all this data and pre-train literally the same-size model from scratch with all of that in mid-training, it will be better. I think that's a lot of what's happening today. There's a real bottleneck stopping us from just continuing to train the same model forever, versus taking all the data from the old model and training a new one from scratch. This is exactly what Charlie was saying: some combination of plasticity and catastrophic forgetting. If you naively train on non-stationary data—because you're adding new data as you go—that messes with the data distribution, so the old stuff just gets forgotten. We don't really have good methods to stop that from happening.

So maybe in the limit you're just bottlenecked by retraining the model from scratch with all this new information?

Yes, which is very expensive. But you're going to do that anyways.

Not necessarily. Maybe eventually, if you have continual learning, you never train a new model—you just have a model that keeps learning and expanding. But there might be some deep technical reason why that's very difficult.

That's the question. I think we have pushed back on how much from-scratch training we need to do. It's definitely possible now to take the pre-trained base and do very good mid-training on top of it, kind of continuously, plus some RL from later checkpoints in the training. That's looking more like continual learning, but it's certainly not the case of taking the most recent model, applying a couple of very small updates, and iteratively never losing anything.

Sorry, I'm a bit confused—isn't this literally what happens during training? During post-training, you have a model that's already gone through so much training, and then you distill some fork that's been further RL'd. Isn't that exactly this?

It's still at a large enough scale, I think, that you're washing out a lot of the noise, and you're not just focused on one distribution—which, as Beren said, is the issue. If you're just focusing on one task—

But in the eventual regime you'd be doing that across billions of deployed instances, learning from all of them at once, so hopefully there's some washing out of noise from that.

Maybe at that scale, yeah. As Charlie was saying, you can definitely do continual mid-training for a long time, and you can roll back to a checkpoint and give it new mid-training data. But you can't do this indefinitely. If you keep continually training the same base forever, it asymptotes at some point—you can't just keep learning new stuff in that base. This is why people end up training new bases; otherwise you would just keep mid-training the same base forever.

This has worked really well. Being able to send informal messages, like I'm texting my editor, and then have the transcript immediately reflect my preferences has been so helpful. Try Grok Bot yourself at x.ai/bot.

The question is how much of AI progress is explained by data progress. That doesn't mean it will necessarily be hard to automate—that's a separate question. Is there some data distribution which, if you trained current architectures on it, would result in a superintelligence that totally dominates human experts across every single field? And are we talking about pre-training plus post-training data, environments as well?

I think the existence of such a distribution is obvious; it's just whether we can create the right environment to get there. In the trivial case, we could just train the model to output the Python file which trains the actual superintelligence—just have that memorized in the weights. Yes, there's probably a ladder of RL environments that could be constructed such that you get an AI researcher at least as good as a human researcher. But the effort to climb each successive rung grows kind of exponentially. Those are the two things you trade off against in how fast we hit that final rung where it's better. I think that's fairly clear.

RL environments and their limits

We're still relatively early in RL environment creation, and there are a lot of asymmetries we exploit to create good environments. One we've discussed before is environments where it's easier to go backwards than forwards: you can define a very complex data-generating process and keep the latent variable hidden from the model. You can generate arbitrarily complex environments, and the model has to do a lot of irreducible token spend and irreducible work to figure out what that data-generating process was.

There are also asymmetries in injecting information from the real world. Anthropic finds a bug through tens of thousands of humans and LLMs combined, and turns that into a very neat environment which a single LLM could theoretically find within a few million tokens. We're cherry-picking all these asymmetries and counting on task-horizon generalization.

But I think this will hit diminishing returns at some point—diminishing returns in how hard it is to create those environments in the first place, because you can't necessarily keep relying on processes where it's easier to go backwards than forwards. You actually have to sit down and construct something that looks like a long enough time horizon with humans, and that's a really complex task to create. Then there are also the compute and time bottlenecks for the agent to actually do those tasks. I think you'll just start seeing this curve flatten out.

Two examples: copying expert behavior vs. too-large gaps

I saw something about how someone fine-tuned the Talkie model, which is only trained on data up to 1930, on modern coding agent data, and it did better than Claude 3 Opus on SWE-bench. So this model with no knowledge of code whatsoever can be fine-tuned on a moderate amount of data and behave better as a coding agent than this much larger pre-trained model—which is pretty crazy. It shows that once you have an example of the right expert behavior, it's surprisingly easy to copy that into a relatively weak model.

A counterexample is a recent paper where they trained a model up to fifth-grade maths, plus primary-school English and so on, so it was a decent language model. They tried to RL it to do late high school and college maths, and the gap was just too large—they couldn't get it to climb at all. But if you did successive rungs of year 7 maths, then year 8 maths, and so on, you could obviously climb to year 12. Again, it's about the distance between the rungs on those ladders and how hard they are to create.

The RL signal problem

This comes back to the RL signal problem. RL is not very good at exploring right now: if the model can't get it in 128 rollouts, it's very unlikely to get signal to progress. This is why in RL we need curricula, whereas in pre-training we don't—that's not a problem for pre-training at all. Pre-training data is different from post-training data.

I imagine humans will be involved less and less as we continue, but that doesn't change the fact that you're bottlenecked on how much signal you can extract from the real world. There is a lot of signal in the world—people doing spreadsheet tasks, legal tasks, and all this sort of stuff. But at the capability frontier of where the models are now, how many bits in the world are actually relevant to improving the model's capabilities? How many new maths problems are being solved that are just beyond the reach of the current models? How many new coding problems are being created or solved beyond their reach? I think that's why the diminishing returns kick in: even the world as a whole is not giving you the bits that are useful for tipping you into the next basin of capability.

Where the signal comes from

I totally agree with this. It's really a question of where the signal is coming from. In pre-training, the signal is already in Common Crawl. For the tasks you care about in pre-training, the problem is not getting signal at all—it's filtering out all the noise, which is quite an automatable process.

But as models get better, as we enter mid-training and post-training, the signal just doesn't exist anywhere in the original data we have. No amount of filtering will get it—there's no hidden proof of a Millennium Prize problem sitting in Common Crawl that we can just filter until we see it. At that point, you have to get bits some other way: either from humans directly, asking them to write out their reasoning; or by creating environments where humans decide what environment should be created and what its objectives are; or some kind of training on the human data that exists in deployment. You have to get the bits from somewhere.

Data vs. architecture in pre-training progress

There's a question of how much of the progress in pre-training is being driven by data. I did this investigation with Jerry Han, a student at Princeton, where we trained all the recipes from 2019 to now pairwise with all the datasets from 2019 to now — training GPT-2 on the newest dataset like Ultra-FineWeb, or Delphi, the newest open-source training recipe, on the Pile or some old dataset. You do the whole grid and see, for a given level of capability, how much less compute it takes across that grid.

The data seems to explain something like a 12.0x compute efficiency gain, while architecture improvements explain something like a 3.7x gain — at a very small scale. To the extent that holds at large scale, meaning most of the pre-training compute efficiency gains come from better data, how much can that continue? Can you keep filtering data more and more, and building more and more synthetic data?

My prior is that the low-hanging fruit is somewhat exhausted. We got the internet as this big block, and the internet isn't necessarily growing at the same rate — nor is all the useful stuff on it. We've probably got a bunch of 0.1% loss drops to go, but definitely not as many as have already occurred.

It's also interesting that you find this cumulative 33x improvement across both. I think it was Epoch or someone who estimated 3x per year since 2019, which would imply something like 3⁷ — over 2,000x. So where's that missing 100x or whatever coming from? That probably gives you a good signal of how much of this is post-training.

I think the explanation has to be that a lot of the compute efficiency gains are scale-dependent, and we're starting at extremely small scale. That raises the question of whether the data-driven or the algorithmic compute efficiency gains have more scale dependence.

We just didn't have enough compute to investigate that. Naively, theoretically, the scale dependence of the architecture is fairly well known and you can fit a straight line to it, whereas I would have no idea how to do that for combining pre-training, post-training, and mid-training data.

Architecture unlocks regimes; data matters more at scale

Funnily enough, I feel like data is actually more important with scale, and architectures are kind of a one-time thing. Saying just an X% efficiency gain is misleading, because what an architecture does is let you reach a qualitatively new regime you couldn't reach with the old one. Within that regime, data is obviously the primary determinant. But if we didn't have even GQA — if we were doing full attention all day — it would be ridiculously expensive to do a million-token context, and because of that we could never use data that actually lives at a million-token context, so we couldn't get those capabilities. If you naively measure "how much does this help at 2K context," where the architecture isn't unlocking anything, the data will look much more important than in some sense it is. It's unclear to me that these things are really just multiplicative gains.

On scale dependence, I think a lot of the mid-training and post-training data we have now actually gets better with scale, because much of it — the very long-context-horizon environment stuff — really requires big models to make use of it. If you try to train a 100-million-parameter model on SWE-bench traces, it's not going to get anywhere; it won't show the improvement you'd get from training a sensibly sized model on it.

It's also hard now because many architecture changes — look at Kimi or DeepSeek — aren't made just with dropping the pre-training loss in mind, but with how the models will be used in the real world. The inference efficiency of having some form of compressed attention in the DeepSeek models isn't necessarily a fundamental trade-off improvement; it's just, "Okay, we're considering how the models are going to be used."

Parameter scaling in an RL-heavy regime

One question for understanding the future is how parameter scaling will go as we move into a more RL-heavy regime. Looking at open-source architectures, parameters have been scaling at roughly 2x per year for frontier open-source models. To the extent that even frontier closed-source models have 100B or 200B active parameters, does that keep 2x-ing year over year? Now that we're in an RL regime where you also want to conserve compute on rollouts — and maybe there's a threshold effect where, once you have enough capacity, increasing parameters arbitrarily doesn't matter as much — how many active parameters will a frontier model have in 2030?

For the next few years, because we're so focused on longer and longer horizon rollouts for RL, where inference efficiency matters a lot, the models aren't necessarily saturated on their ability to do that — the bottleneck is still the environments. So we might see a bit of a plateau. I have a feeling that Mythos and the GPT models are much smaller than the 10-trillion-parameter range people talk about; even naively comparing them to open-source models, you can probably back out that conclusion. Probably for the next few years I wouldn't imagine huge growth in parameter counts. But there are so many things to trade off: you decide the size of your model based on how much pre-training data you have, and then on the difficulty of the RL environments you have to train on.

Ideally you want to reach the point where you get a decent pass@1 on the hardest environments you have. It wouldn't make sense to make a bigger model pass there, because then you're paying far more for inference than you need to. So a lot depends on how quickly Mercor and the in-house teams can scale up the complexity of the RL environments they're training on. I'd expect models to keep getting bigger simply because people are scaling up compute and GPUs are getting bigger, but exactly how much bigger depends on the scaling laws in non-obvious ways.

One thing I think is that data efficiency will be a bigger driver than compute efficiency of the exact architectures people use, now that we're running low on high-quality pre-training data. That might affect how sparse you want to make the model. We also don't understand sparsity that well — parameters are a different resource than active parameters. Sparsity has definitely increased a bit, but it's not clear it will keep increasing without bound; there might be some sweet spot. There's an argument that sparsity should make data efficiency worse, because you might have to learn the same thing on multiple experts, though that's debatable. We don't have a good enough theory of scaling laws to really understand why sparsity helps, how much it helps, or whether it plateaus at some level of sparsity.

"Sorry, can you spell out exactly what the implication of data efficiency would be on parameters? It sounds like you'd say there should be less sparsity, but what are the other implications on parameter scaling?"

Just that with the scaling law, you're not trying to optimize compute efficiency. Each architectural choice gives you a different scaling law. Traditionally you'd look at some kind of envelope based on compute — performance versus compute, taking the envelope of the best models. But if we're making that decision based on data — assuming we can spend a lot of compute, so data is on the x-axis instead of compute — then we get a different set of optima, a different set of models on that frontier.

I also don't think we've necessarily doubled model size every year for the last few years. People have been training 1 trillion parameter models for at least a few years — there was even an open source one called Falcon. Liam from Periodic Labs, I think, posted yesterday on Twitter about how an early experiment was training a 1 trillion parameter model that was very, very sparse. That was what they did before OpenAI, at Google — the Switch Transformer. It was very, very good at knowledge but terrible at reasoning because it was so sparse. It feels like we've been playing in the 100 billion to 2 trillion parameter range for a while; it certainly hasn't been a nice linear increase.

I think there are two things going on. As Charlie was saying, inference efficiency is super important for RL rollouts, which will really push down active parameters. Total parameters also depend a lot on the hardware: you need very high memory bandwidth and VRAM to actually serve multi-trillion parameter models. Right now people are still using a lot of H100s; as everyone moves to GBs and then Vera Rubins, we'll get more ability to scale, serve, and do large RL inference at larger scales.

The data question is interesting, because naively, larger models are much more sample efficient on the actual data points. Even if you're not saturating the model, it's still better to go bigger, because larger models generalize better and reach a better loss for the same amount of data. Right now we have a lot of data, so that's not the constraint — compute is, which is why we have smaller, very inference-efficient models. But if compute stops being the bottleneck, it might come back to larger, undersaturated models that have this generalization ability because they're much larger.

If you just look at the basic Chinchilla scaling law and maximize parameters, it actually decreases the amount of data you need to reach the same loss only very little — if you go to infinity on parameters, the data needed goes down less than 10X, just because of the nature of the power law. But we're now on the way-too-much-data side of the Chinchilla laws: we currently over-train models according to Chinchilla. So we could easily get back to a point where, as we run out of data, we move back to the Chinchilla optimal point, or even a bit to the under-training side. But surely, even with these new chips coming online, we'll be so compute bottlenecked for the next few years that that won't necessarily happen.

This really depends on your ratio of training to inference compute. If you're super bottlenecked on data, not compute, you should go bigger; if you're super bottlenecked on compute, you should always go smaller. You can also use computer-generated synthetic data, so it's one of these very hard things to predict.

Part of the reason it took people so long to figure out the scaling laws in the first place is that if you don't get all these things right, you don't get such a clean relationship. The beautiful straight lines on graphs hide a lot of complexity — you have to scale every hyperparameter the right way, or parameterize your optimizer so it scales and you don't have to change hyperparameters as model size changes. Bugs have their own clean scaling laws too: like Kaplan forgetting the cosine annealing thing, or not considering embedding parameters, which messed up the estimate at smaller models because embedding parameters are a decent fraction of the model.

A bit on RL. A year ago, a lot of people

A year ago, many people argued that RL would not scale well for models. John wrote a research paper pointing out that models learn roughly one bit per RL episode — whether the answer was right or wrong. I wrote blog posts earlier this year arguing it was even worse: when the pass rate is low and the model is very unlikely to get the answer right, it learns almost nothing from an episode. Yet today's models look pretty smart, seemingly the result of scaling up RL. Beren, in a post a few weeks ago you tried to explain what's going on — why has RL been more successful than one would naively expect?

I think the success of RL comes down to several things. First, what is slightly underestimated is mid-training: a lot of what we see as RL successes actually comes from very good mid-training data, where we're essentially doing pre-training on synthetic reasoning data and environments that warm-start the model for RL. This often takes the model almost 80% of the way to the final RL checkpoint. RL then just tweaks the policy — which is why it doesn't need as many bits as you'd naively think. It doesn't have to learn these behaviors from scratch; it needs just a few bits from these episodes, which you do get.

The other point from my blog is that these bits are extremely high-signal compared to regular pre-training, which is why you need RL at all rather than just SFT on successful reasoning traces — it's exactly the bits about how to get the answer right.

But it's not quite how you'd think of it. In SFT you have a trace — say, math reasoning followed by the answer at the end. The bit is still there; you still SFT on the answer token. What's important is that the objective ignores all the other bits. In SFT, you have to match the exact reasoning tokens, so you're getting too many bits about how that other model reasons. With RL, you only get the one bit, so the signal isn't drowned out by the noise of all the other bits. It's a dramatic increase in the signal-to-noise ratio during training, which is why RL is so efficient in terms of steps.

There's been a lot of debate about what RL does to the model versus mid-training or SFT. Everyone talks about how pass@1 goes up but pass@256 goes down — rare correct reasoning traces get down-weighted by gradient signal from easier traces. I think the simple way to view RL now is: if you have enough compute to sample a large enough group size, such that the probability of getting a bunch of correct answers is past some not insignificant threshold, then those traces will be up-weighted. And to Beren's point, mid-training and more pre-training — the pass@1 starting point for RL — scales with the log of pre-training tokens.

Can I ask some very basic questions? That answer makes sense, and maybe there's empirical research showing this is what's happening. But when I just look at the models themselves, they've qualitatively gotten so much more capable. Maybe there's no inherent contradiction, but how do we square the relatively small impact this view implies RL would have with the actual qualitative gains in capabilities?

One thing to point out: it doesn't necessarily imply RL has a small effect. Even with a few bits and only a small change in parameters, the impact on function space — the input-to-output mapping the model learns — can still be dramatic. Even one bit can rule out half the hypothesis space, which is huge. So starting from a really good point, small amounts of RL don't necessarily mean small impacts on behavior.

I think it comes down to two things. First, everyone hoped RL would generalize reasoning horizontally across all domains. We didn't really get that: training on math doesn't necessarily make you a great coder — you do have to do RL on code environments. What we did get is horizon generalization: models learned to use more tokens for longer and still make progress on a task. You can train on environments that get longer and longer, then put the model into a completely new environment. It may not have generalized the reasoning patterns for that environment, but it has generalized the ability to keep going for longer, which correlates with success. A paper called EdgeBench showed that the length of time models can work for is doubling every three months — clear evidence of generalization.

The final way to think about it is the idea of quanta from pre-training. The pre-training loss curve is very smooth, but the model is actually learning discrete tasks, with emergent phase transitions — it didn't have induction heads, now it does. There are tens of millions, probably hundreds of millions of these; averaged together they produce the smooth loss curve. Something similar happens with RL. There's a slow outer loop: we train a model, RL it, then in the next training iteration dump a bunch of synthetic reasoning traces into the mid-training data. We're hitting quanta for many different tasks, and on an individual task level it can look like a phase transition — suddenly going from a 0.5% pass rate to a 90% pass rate on a particular finance or Excel task. But averaged together, plus the horizon generalization, you get qualitatively better models.

I'd add that RL does generalize a bit — you get some transfer between math and code, or puzzles and math. Also, the sheer number of environments being targeted is vastly greater. Two years ago, if you tried some everyday task, the labs wouldn't care and wouldn't train the model for it. Now the coverage is much broader, with many environments targeting specific things.

Earlier we discussed RL causing entropy collapse—concentrating probability on solutions the base model already had and producing relatively sparse policy updates. But there's another story about RL: going back to the Atari games and then AlphaGo's move 37, the super creative move. Because it was never initialized on human data, it can think in ways humans don't and come up with extremely creative solutions. Should we expect RL on LLMs to produce something like move 37—creativity beyond human creativity, from a de novo initialization of intelligence?

A couple of things here. First, AlphaGo uses MCTS, which obviously does more exploration than regular policy gradients. But I also don't think RL necessarily reduces creativity. This is qualitative, but look at the OpenAI–Hugging Face incident: those models came up with multiple zero-days at a time to break out of the sandbox. That's clearly some level of move 37 creativity already, arising from the general generalization properties of LLMs. It's definitely not the case that RL is totally destroying entropy, especially on long horizons.

One thing people call creativity is just solving hard search problems. Move 37 is an example, or writing a poem that satisfies a ton of different constraints—something AI will obviously be extremely good at, if trained for it.

But there's another sense in which the diversity of models' outputs is much lower after RL, and they develop tics. Even though the models seem good at writing, distributional analysis shows they reuse certain themes and the same character names all the time. You don't get the diversity of human authors—you get one really good style. That kind of diversity has definitely been cut down a lot by RL.

Related to distillation: so many people are distilling, mostly from Claude, that all the open-weight models write the same way as Claude and share its tics. This emerging monoculture seems concerning to me. But I don't think this is fundamental to RL as a method, and the same goes for distillation—with distillation you're just training on the data; if your data isn't broad, that doesn't mean the training method itself is wrong. It's a problem with the data.

A lot of RL entropy collapse, for instance, comes from exploiting fairly simple verifiers when you don't have a huge diversity of environments. Writing is presumably graded by some judge; the judge has specific tics, and the model learns to reward-hack the judge, which is why it collapses. But that's really a problem with the judge, not with RL in general.

The speaker asks for timelines on a series of predictions, starting with models that can be hired as a drop-in remote worker for all kinds of white-collar work — not just coding, but video editing, law, paralegal, and so on. It would be a literal remote worker with full computer use, a month of seamless learning and operation, executing complex projects that require interacting with other people — everything a human worker could do over a month.

If you mandate that it use a browser rather than the firm making information programmatically accessible, maybe a couple of years. But if it can send Slack messages and do all this stuff without being browser-based, around a year — and maybe three years for the full generality. To Charlie's point, though, people will end up making their organizations easier for AIs to use, so you get 80–90% of the way there before that. The difference between one year and three years is small in that light.

There will be a long tail of miscellaneous things some human can do that will take the models quite a while. Asked whether that means computer stuff or basic cognitive capabilities, the answer is that it really comes down to how quickly we can solve online learning, and whether we can get 80–90% of the way there with compaction and writing files to yourself. That's the big uncertainty.

One example the model wouldn't be good at: having to yell at someone at work, or really push someone to get something done. The model just won't do that — it will be too nice.

Another speaker notes there's wide variation in the quality of human remote workers: hiring someone off Upwork for a software engineering project involves huge variation, and it's often hard to get them to do a good job or pay attention to feedback. In some cases the pre-AI version of this was worse than what existing AI can already deliver. It might end up being complicated, because we already have this for some not-so-high-quality work, but we're clearly not matching human level in certain higher-quality forms of it. Still, they basically agree with Charlie and Beren that some version of this — the right form factor, doing some things really well and others not so well, improving from there — arrives in a year or so.

We shift the goalposts based on the very long tail all the time. As an example: this year, the speaker literally just told Codex to gather everything needed and send it to the accountant for their taxes — a massive list of things requiring clicking through and downloading on computers. It did it, and it was perfect. A lot of this it can already do.

10x productivity uplift

Next question: a 10x total productivity uplift — if a breakthrough takes you a year now, you make one every month. One speaker refuses to give a scalar, noting we might already be past that in some types of work, such as certain types of math.

Clarified for AI researchers specifically: how much are AI researchers sped up? One answer: somewhere between 5–10 years. Another is surprised that's longer than for a general remote worker — then realizes they must have different definitions. The remote-worker question meant normal white-collar work over a month, diverging a bit past two months: a very competent white-collar worker, but not necessarily a super creative researcher. For the 10x researcher uplift: two years.

Beren can see that, because right now it's already more than 10x for coding — so if the AI can do even one or two loops of experimental feedback, that would be massive already. Plugged into a naive model of how much AI progress comes from AI researchers, a 10x increase in their productivity means a radically accelerated pace of AI progress starting two years from now. Progress stops being bottlenecked on AI researchers' ability to run small experiments and gets bottlenecked on other things — but it happens 10x faster, which is a huge deal, and it also makes the next milestone, a 100x speedup, happen sooner.

Asked for the crux of his longer estimate, Beren points to his own capacity to absorb information and make the Bayesian-optimal decision on the next experiment. But if some of that can be delegated — the AI is becoming decent at deciding what experiment to run next based on results — then an AI that can run two or three experiments in a row without crashing is already a big uplift.

Dominating all cognitive work

Final question: an AI that dominates top human experts across every single field of work that can be done over a computer — not just AI research but all cognitive work, not just short-horizon work; if something takes three years, the AI still does better than humans. That's basically ASI? Yes. Answer: 3–4 years.

The reaction: that doesn't seem wrong, but — AI is obviously getting more attention. It's one of the harder things, but a lot of energy is being put into it.

It's also not one of the hardest things for AI, because it involves a lot of code and math, which models are really good at. Tasks involving 3D, spatial, and physical stuff will take a little longer. Mechanical engineering, for example, isn't getting the most attention right now, so that might take longer. It also includes fields where there's relatively little data because of the nature of the field, and the AI has to learn that data on the fly—for example, becoming superhuman as an engineer at TSMC. You'd have to assume you can give the AI the same onboarding material a human gets. Something also has to be solved about longer-horizon learning. I'd say 5 to 10 years.

So basically, you think automating AI research is ASI-complete or something?

Yeah, I think so. There are so many things in the world where, even with some memory system external to the model, and even if context length grows a bit, there are fundamentally tasks where you'd need more than a million-token context window today, even if you could research the information or write notes yourself.

I kind of agree on the 5-year range, at least for the stuff labs are focusing on. But I think there's going to be a long tail of things the AI could theoretically go out and learn about, but no one has bothered to do it and the compute hasn't been allocated to that. So that might take longer for literally every single human expert.

Sorry—by this I also meant the ability to learn a new domain as fast as a human.

I think that's not necessarily necessary, because the AI will have vastly greater experience than any human.

Thanks so much for doing this, guys. This was a great format for getting different experts to disagree, debate, and discuss things together—very productive.

Thanks for having us.