Beren Millidge, John Schulman and Charlie O’Neill expect AI to do more of the work involved in building better models. They disagree about how soon it could take over that work, and whether today’s training methods can produce a system that chooses its own research goals and learns reliably from experience.
In a September 2026 conversation with Dwarkesh Patel, their estimates for AI surpassing top human experts across computer-based work ranged from three or four years to five or ten. Those estimates depended on different assumptions about learning, research judgement and the work they counted as fully automated.
Millidge is the CTO of Zyphra. Schulman is the chief scientist at Thinking Machines and previously co-founded OpenAI, where he led work on reinforcement learning from human feedback for ChatGPT. O’Neill heads model training at Baseten.
They discussed recursive self-improvement, or RSI: AI systems helping to build more capable AI systems, which then help build the next generation. The disputed part is how far that process could proceed without people directing it.
Why better benchmarks might not be enough
Patel opened with a hypothetical. Suppose it is 2036 and superintelligent AI has not transformed the world. Excluding war, regulation and other political shocks, what technical problem might explain that?
Millidge suggested that models might keep improving at tests without becoming as useful outside them as people expect. An AI can excel at chess or competition mathematics while struggling with other tasks. He connected this to Moravec’s paradox, the observation that abilities people find difficult can be easier for computers than abilities people take for granted.
In this scenario, models become extremely good at the tasks researchers can put into tests and training environments. They still struggle to use those skills in unfamiliar real situations, learn how to learn new tasks, or retain what they learn after deployment. Millidge considers this unlikely, because he already sees some skills transferring beyond their training tasks.
Schulman pointed to weaknesses in judgement and self-checking. A model can initially impress its users, then reveal limitations after a month of use. He thinks this cycle could repeat more times than researchers expect. Writing much more code does not make a person equally more productive if they still have to decide what to build and check whether it works.
Patel offered chess engines as another way to understand progress. Their playing strength rose steadily for decades, but their relationship to human players changed sharply when they passed through the range of human ability. A steady improvement could therefore have much larger consequences once AI starts consistently outperforming people.
Millidge agreed that, for Patel’s hypothetical to hold, technical progress might have to slow just before that crossing. He thought substantial regulation was a more likely reason for a relatively unchanged world in 2036, although Patel had excluded it from the question.
Whether today’s models can discover a different way to learn
O’Neill questioned how close today’s systems are to the best learning system that could run on a chip. Current models commonly combine transformers, a type of neural network that processes relationships within data, with reinforcement learning, which rewards successful behaviour.
If a model became even slightly better than the best human AI researcher, researchers could run many copies in parallel. That is the argument for rapid self-improvement. O’Neill’s concern is whether a model trained with today’s methods could discover a substantially different method if further progress required one.
He compared this with Moore’s law, the long-running increase in the number of transistors on chips. The steady trend depended on engineers repeatedly inventing new techniques. He sees AI progress similarly: reinforcement learning helped extend progress as gains from pre-training, the initial large-scale training stage, became harder to obtain.
Another improvement might fit naturally into the existing approach. A more radical change could require researchers to rethink neural networks or gradient descent, the method used to adjust a model during training. O’Neill does not know whether scaling today’s models would let them discover such a change.
Choosing an experiment is different from running it
Anthropic’s later Opus 5.5 evaluation offers a concrete test of that distinction in the article on automating AI research.
O’Neill distinguished research with a clear target from research in which someone first has to decide what target matters.
A system might become much faster at reducing a model’s prediction errors or improving a known training method. Choosing a new way to approach intelligence is less clearly defined. Researchers cannot always specify a score that tells an AI whether it has found the right direction.
Patel imagined spending large amounts of AI computation analysing an experiment before and after running it. Schulman thought there was considerable room for improvement this way. Models could develop theories, examine existing results and design smaller experiments that help predict what will happen at a larger scale.
O’Neill gave examples of mistakes an AI might catch earlier. He argued that closer analysis of the Kaplan scaling laws, which described how model performance changes with training resources, could have exposed problems with how intermediate training checkpoints were compared. Those comparisons did not properly account for the learning-rate schedule, which changes how large an adjustment the model makes during training. In his view, spotting this could have saved a year or two of work.
He also mentioned muP, a method for transferring training settings between models of different sizes, and the relationship between learning rate and model width. These were examples of gains he thought researchers could obtain by examining existing evidence more carefully.
O’Neill could imagine a tenfold speedup when the goal was already clear. He did not think that established an equivalent improvement in choosing the goal. His point was that reasoning can help interpret available evidence, but cannot replace every observation or experiment.
Millidge described the requirement for an independent research loop: the AI would have to propose a goal, pursue it, evaluate the result and choose another goal, repeatedly, without the process going wrong.
Why people may still have to define the goals
Asked which part of AI research humans would do longest, Millidge chose asking the right questions. He finds models more capable at coding an experiment than at proposing a worthwhile one. Their suggestions often consist of many small changes.
O’Neill pointed to Alec Radford’s decision to train models to predict the next piece of text across a wide range of data. Even after that approach worked, researchers needed evidence that it would improve predictably with scale before committing much larger resources.
Schulman thought people would remain responsible for deciding what AI should do. A post-training team has to specify what helpful behaviour means in different situations. Documents such as model specifications and constitutions record those choices.
He separated this from the technical work of making a model follow the chosen objective. AI might automate more of that work while people continued deciding what behaviour they wanted. This was his view of the human role in alignment, the work of making AI behave as intended.
What reinforcement learning changes
Patel asked why reinforcement learning had produced so much progress despite apparently providing little information in each attempt. With a simple right-or-wrong reward, a model gets a limited signal. If it almost never succeeds, it has very little useful feedback to learn from.
Millidge said the model often knows much of what it needs before reinforcement learning begins. Mid-training, an additional training stage using selected data such as AI-generated reasoning examples, gives it a strong starting point. He estimated that this often takes a model roughly 80% of the way towards its final reinforcement-learning performance. That was his assessment, rather than a universal measurement across laboratories.
He also explained why a small reward signal can be useful. Supervised fine-tuning teaches a model to reproduce an example answer and its reasoning, step by step. This can make it copy details of the example that do not help solve the problem. Reinforcement learning can instead reward getting the answer right without requiring the model to reproduce a particular explanation.
O’Neill said training on mathematics had not made models equally strong at coding. Researchers still needed coding environments. What seemed to transfer more readily was persistence: models learned to keep working, use more steps and make progress over longer tasks. Millidge added that some transfer between subjects does occur, and laboratories now train models on a much wider range of tasks.
O’Neill also argued that smooth overall progress can conceal abrupt improvements on individual tasks. A model may go from rarely succeeding at one task to succeeding most of the time. Averaging many such changes makes the overall improvement look gradual. He used a jump from a 0.5% success rate to 90% as an illustration, not as a reported result for a named test.
Learning from use without forgetting earlier skills
Patel asked why the computation spent using models did not more directly improve them. He estimated that inference, running an already trained model, accounted for about half of computation. He imagined a model eventually learning from the combined experience of its many deployed copies.
Millidge said a slower version already exists. Researchers can select and process deployment records, then use them to train a later model. Distillation works similarly: one model learns from examples produced by another.
O’Neill pointed to Cursor’s Composer and Harvey’s legal agents as examples of shorter feedback cycles. He described Cursor using feedback from deployed models to make reinforcement-learning updates, and releasing an updated version every five hours if it improved on CursorBench, the company’s internal test. Humans still chose the feedback signals and built the training environments.
Schulman cautioned that a simple signal, such as whether a user accepts an edit, may reward the wrong behaviour. A model could improve that score without becoming genuinely more useful.
Repeated small updates to one model remain difficult. O’Neill described a model learning from limited data for a particular customer. After hundreds of updates, it can lose information it learned earlier and become worse at general tasks. This is catastrophic forgetting.
On-policy distillation, in which a model learns from corrections to answers it generates itself, can delay that problem, according to O’Neill. He said it eventually encounters similar limitations. Reinforcement learning makes smaller changes, which can help preserve existing skills, but also limits how much new information it can add.
Learning a skill and learning a specific fact are different problems. O’Neill said reinforcement learning is better at improving behaviour than teaching details such as which employee handles a process at a particular law firm. Researchers may have to spend substantial computation building environments that teach those facts.
Millidge considered the difficulty mainly a problem with training techniques. A fresh model of the same size, trained with all the relevant information, can perform better than one that receives repeated updates. He said this helps explain why laboratories keep training new base models rather than updating one indefinitely. O’Neill allowed that both model capacity and training methods could matter.
Who controls deployment data
Schulman said business incentives would also affect whether a model could learn from every customer. Companies may not want their experience to improve a provider’s shared model and benefit competitors.
O’Neill expected companies to favour separate modules. A LoRA is a relatively small set of trained adjustments that can specialise a model without changing all its underlying parameters. Other approaches compress information from a long context, the material a model can use while working, into a smaller reusable form. He mentioned cartridges, which compress cached representations of that information, and work on linear attention that aims to handle long inputs more efficiently.
Millidge suggested that learning could become faster in stages. Companies could specialise a model for a deployment, collect examples, and use permitted data in the next generation. As the retraining cycle shortened from months to weeks or days, frequent retraining might achieve much of what people mean by continual learning. He presented this as a possible route, not a solved method.
How much better data can do
Patel described research with Jerry Han that compared combinations of public training recipes and datasets representing 2019 through 2025. At the small scale they tested, better data produced about a twelvefold improvement in compute efficiency, compared with about 3.7-fold from better model recipes. Compute efficiency here means needing less computation to reach a given level of test performance. The model changes included more than architecture alone, such as changes to training settings and optimisation.
Patel and the guests cautioned against applying those figures directly to much larger models.
Millidge said architecture can make a different kind of training possible. Grouped-query attention, which reduces the memory needed for parts of a model’s attention computation, makes very long inputs cheaper to handle. Without it, some million-token inputs would be too expensive to use. A small-scale comparison with short inputs can miss that benefit.
O’Neill thought many of the easy gains from filtering internet data had already been made. Useful online material was not growing as quickly as computation. Schulman expected learning efficiently from limited data to become more important as high-quality pre-training material became scarce.
That could affect how models use sparse networks, in which only some of their parameters are active for a particular input. Schulman said researchers still lacked a good explanation of why sparsity helps as much as it does. One concern was that different parts of a model might have to learn the same information separately, making less efficient use of the data.
The estimates depend on the task
At the end, Patel asked when AI could act as a remote worker for a month, completing projects and interacting with other people.
O’Neill estimated about a year if a company made its information accessible through software interfaces and messaging tools, or a couple of years if the model had to work through a browser. Millidge suggested about three years for the full range of tasks, with useful partial versions arriving earlier. Schulman expected an uneven but useful version in roughly a year. None described a worker that would immediately match the best humans at everything.
For a tenfold improvement in AI researchers’ overall productivity, Schulman suggested two years. Millidge could see that happening if models could run and interpret several experiments in succession. O’Neill estimated five to ten years. He distinguished ordinary month-long office work from open-ended research and longer projects.
For AI surpassing top experts across all work that could be done on a computer, Schulman estimated three or four years, O’Neill five to ten, and Millidge roughly five for areas laboratories were actively targeting, with other areas potentially taking longer.
Schulman’s answer assumed the AI could receive the same onboarding information as a person and that researchers would solve important problems in longer-term learning. Millidge remained uncertain about tasks that laboratories had not studied or trained for. The estimates were the speakers’ forecasts from September 2026, not established dates for those capabilities.