John Schulman on dead ends, scaling RL, and building research institutions
Cursor · 2025-12-17
💡 Quick Take
1. You could build ChatGPT-level AI much earlier, maybe 2018-2019, with fewer people and less compute if you had full hindsight and the right recipe.
2. Early OpenAI was more informal and "ragtag," like an academic group, with researchers working on individual projects before shifting towards bigger, more coordinated engineering efforts.
3. Projects like "Universe" and robotics were early, ambitious but ultimately unsuccessful, dead ends for OpenAI, though they built valuable capacity and trained people.
4. Dota was one of OpenAI's earliest truly successful "big big projects" that combined significant ML systems engineering with RL research.
5. Research management styles vary: hands-on managers for goal-oriented work with less experienced teams, and hands-off managers for exploratory research with experienced individuals.
6. OpenAI drew inspiration more from previous work experiences (Google Brain, DeepMind) than from historical labs like Bell Labs or Xerox PARC.
7. Early OpenAI and Thinking Machines share similarities in having multiple parallel projects shaping the company's vision, but Thinking Machines operates in a faster-paced, competitive field.
8. Value functions aren't currently very helpful in popular RL settings like RLHF or on short time horizons, possibly due to insufficient variance reduction.
9. Continual learning might be solved by improving in-context learning and long-context abilities, potentially with LoRA or parameter fine-tuning for specific memory types.
10. We might not need entirely new ideas beyond context windows and fine-tuning to solve continual learning; scaling models could improve metrics, or new methods could offer better scaling laws.
11. Generalization remains a question mark for AGI; while in-context learning shows high sample efficiency, some training still requires much more data than humans.
12. Co-training generators and verifiers (like in GANs) and multi-agent training/games offer promising avenues for AI development and alignment.
13. AI is heavily used for coding (Cursor, Cloud Code) and research, including literature searches and fleshing out ideas.
14. Effective research involves both idea formation (coffee shops, notebooks) and execution (coding, reading docs/code, advising).
15. The core advice from a 2020 research blog post (goal-directed research, notebooks, building taste) still holds, with keeping a lab notebook being even more crucial now for LLM feedback.
16. Incorporating LLMs into research workflows is key, but understanding code line-by-line remains vital for research, unlike in some other software engineering domains.
17. The rate of consequential big idea generation in ML might be constant, but the standards for experimental rigor and overall quality have increased significantly with more researchers.
18. Internal company research reporting can be more accurate in conclusions but less thorough in write-ups and baseline comparisons than academic publishing.
19. The influx of researchers has led to higher bars for entry and a greater emphasis on engineering skills over pure research taste and exploratory abilities.
20. The future of RL research will likely see ideas cycling back into fashion, with offline RL and sim-to-real techniques (like in robotics) being particularly interesting.
21. Coordination between leading AI labs on safety is plausible due to common viewpoints, but personal rivalries could be a hurdle.
22. AGI timelines are likely underestimated; engineers and researchers tend to be overly optimistic, and historical analogies like self-driving cars suggest longer development cycles.
23. Tinker is a low-level fine-tuning API from Thinking Machines that abstracts away GPU and distributed systems complexities, aimed at sophisticated ML practitioners.
24. Tinker's ambition is for future AI companies to build on its infrastructure rather than developing their own, eventually becoming more user-friendly for a wider audience.
25. Thinking Machines plans to release new models and enhance Tinker with multimodal capabilities and larger job scaling in the near future.
📊 Detailed Explanation
1. You could build ChatGPT-level AI much earlier, maybe 2018-2019, with fewer people and less compute if you had full hindsight and the right recipe. This is a fascinating thought experiment! The idea is that with all the knowledge we have now about pre-training, post-training, and fine-tuning techniques, the original OpenAI team could have essentially reverse-engineered ChatGPT much sooner. They mention that nanoGPT, a project by one person, shows you can do it with less compute. The key here is "full hindsight" – knowing exactly what worked and what didn't. They believe that with clever post-training and a well-constructed fine-tuning dataset, even a smaller model could achieve impressive results, potentially reaching a GPT-3.5 level back in 2018-2019 with just a few talented people working for about a year. This highlights how much of AI development is iterative and how knowing the destination makes the journey much faster.
2. Early OpenAI was more informal and "ragtag," like an academic group, with researchers working on individual projects before shifting towards bigger, more coordinated engineering efforts. It's easy to forget that even massive tech companies start small and scrappy! The transcript paints a picture of early OpenAI as a collective of researchers driven by their own interests, forming small groups of one to three people to work on projects that might lead to papers or blog posts. This academic-like atmosphere, while fostering exploration, eventually evolved. They were influenced by DeepMind's success with large-scale projects like AlphaGo and recognized the potential to "go a lot further by doing serious engineering and putting together bigger groups of people." This marks a shift from pure individual exploration to a more structured, project-driven approach.
3. Projects like "Universe" and robotics were early, ambitious but ultimately unsuccessful, dead ends for OpenAI, though they built valuable capacity and trained people. Not every ambitious idea pans out, and that's okay! "Universe" aimed to create a vast dataset of RL environments (video games, web navigation) to train a general RL agent. While the idea was "deeply correct," it was "way too early," lacking necessary prerequisites. The system was unwieldy and not great for RL experiments, leading to models that didn't generalize well. Similarly, robotics projects were somewhat of a "dead end." However, the crucial takeaway is that these "failures" weren't wasted. They were instrumental in building OpenAI's capacity for large-scale engineering and research, and importantly, in training a lot of people in these complex domains.
4. Dota was one of OpenAI's earliest truly successful "big big projects" that combined significant ML systems engineering with RL research. This is a great example of a project that *did* work! Dota, the multiplayer online battle arena game, became a major focus for OpenAI. It wasn't just about the RL research itself; it involved substantial ML systems work. This included building the "environment infrastructure" – how to hook into and programmatically control the game – and the "training system" for large-scale, parallel training, likely using techniques like asynchronous RL. It demonstrates the blend of complex engineering and cutting-edge research needed for breakthrough AI achievements.
5. Research management styles vary: hands-on managers for goal-oriented work with less experienced teams, and hands-off managers for exploratory research with experienced individuals. Managing brilliant minds is tricky! The transcript highlights two effective management models. The "hands-on" approach involves the manager being deeply involved in the code, providing detailed technical feedback, and being very goal-oriented. This works well when the team is less experienced or the project has specific objectives. The "hands-off" approach, where the manager acts as a sounding board, offers career advice, and focuses on keeping people motivated, is better suited for exploratory research with highly experienced individuals who can largely self-direct.
6. OpenAI drew inspiration more from previous work experiences (Google Brain, DeepMind) than from historical labs like Bell Labs or Xerox PARC. It's interesting how influence flows! While historical research labs like Bell Labs and Xerox PARC are legendary, the speaker suggests that for many at early OpenAI, the more direct influences came from their prior experiences at places like Google Brain and DeepMind. This makes sense – people tend to replicate or adapt what they've seen work in their immediate professional past. While there might have been discussions about projects like the Manhattan Project, there wasn't a deliberate, deep dive into analyzing the structures of past successful research institutions.
7. Early OpenAI and Thinking Machines share similarities in having multiple parallel projects shaping the company's vision, but Thinking Machines operates in a faster-paced, competitive field. There's a parallel here between how new ventures define themselves. Both early OpenAI and Thinking Machines started with multiple different projects running concurrently, allowing the company's vision to emerge organically as these projects took shape. However, the context is vastly different. Thinking Machines is operating in a field that's moving at lightning speed, with intense competition. This creates pressure to "catch up" to the state-of-the-art, whereas early OpenAI had more of a "peacetime" feel, allowing for more purely exploratory work without the same immediate competitive urgency.
8. Value functions aren't currently very helpful in popular RL settings like RLHF or on short time horizons, possibly due to insufficient variance reduction. This is a bit of a puzzle in RL! Value functions are typically used for variance reduction, which should make learning more stable. However, in current popular RL applications like Reinforcement Learning from Human Feedback (RLHF) or tasks with relatively short time horizons (even if they involve many tokens), value functions just don't seem to provide much benefit. The speaker can't definitively say *why* this is the case, but it's a noted phenomenon that might change as tasks and methods evolve.
9. Continual learning might be solved by improving in-context learning and long-context abilities, potentially with LoRA or parameter fine-tuning for specific memory types. Solving continual learning – the ability for AI to learn new things without forgetting old ones – is a huge challenge. The speaker anticipates that advancements in how models handle context (in-context learning and longer context windows) will be crucial. They also see a role for parameter-efficient fine-tuning methods like LoRA, which could be particularly good for absorbing a lot of knowledge and forming certain types of memory, especially those requiring significant capacity.
10. We might not need entirely new ideas beyond context windows and fine-tuning to solve continual learning; scaling models could improve metrics, or new methods could offer better scaling laws. This is a nuanced point about the future of AI development. While scaling current models might lead to improvements in continual learning metrics, it's also possible that entirely new methodologies could unlock much faster or more efficient learning. The speaker suggests that new approaches could lead to different "scaling laws," meaning you might get a bigger bang for your buck in terms of effective compute or learning speed. It's a balance between iterating on existing successful paradigms and exploring novel ones.
11. Generalization remains a question mark for AGI; while in-context learning shows high sample efficiency, some training still requires much more data than humans. True AGI needs to generalize across a vast range of knowledge. While LLMs impress with their in-context learning capabilities, matching human sample efficiency in some areas, there are still significant gaps. Certain training paradigms require vastly more data than it takes humans to learn the same concept. Models can be brittle in ways humans aren't, and humans have evolved to operate over much longer time scales with sophisticated self-correction mechanisms. Whether these differences are temporary or fundamental is still an open question.
12. Co-training generators and verifiers (like in GANs) and multi-agent training/games offer promising avenues for AI development and alignment. Looking back to older ideas can be fruitful! The speaker is enthusiastic about co-training generative models with verifiers (or "judges"). This creates a virtuous cycle where a better reasoning model can train a better generative model, and vice-versa. They also champion multi-agent training and game theory, noting that games provide automatic curricula and can be designed so that their equilibrium solutions involve solving complex problems. This is seen as a powerful tool for both capability development and alignment.
13. AI is heavily used for coding (Cursor, Cloud Code) and research, including literature searches and fleshing out ideas. AI is no longer just a research topic; it's a daily tool! The speaker uses AI extensively for coding, mentioning tools like Cursor and Cloud Code. Beyond that, AI models are indispensable for research. They're used for rapid literature searches, fleshing out vague ideas into more concrete paragraphs, and finding open-source libraries. This drastically speeds up the research process, which used to involve much more manual searching and iteration.
14. Effective research involves both idea formation (coffee shops, notebooks) and execution (coding, reading docs/code, advising). Research isn't just about staring at screens. The speaker describes a personal workflow that balances different modes. "Idea formation" happens in environments like coffee shops, where the ambient buzz can help focus thoughts and jot down initial concepts. "Execution" mode, however, involves more focused work: coding, diving deep into documentation and colleagues' messages, reviewing plots, and advising others. This highlights the multifaceted nature of productive research.
15. The core advice from a 2020 research blog post (goal-directed research, notebooks, building taste) still holds, with keeping a lab notebook being even more crucial now for LLM feedback. Good advice stands the test of time! The speaker confirms that fundamental research principles like having clear goals, diligently keeping a research notebook, and cultivating taste by reading widely are still highly relevant. In fact, the lab notebook is *more* critical now because it can be fed into LLMs to get valuable feedback, making the documentation process even more impactful.
16. Incorporating LLMs into research workflows is key, but understanding code line-by-line remains vital for research, unlike in some other software engineering domains. This is a crucial distinction for researchers. While LLMs can accelerate coding by generating large amounts of code, for research, a deep, granular understanding of every line of code is often essential. This isn't necessarily true for all software engineering tasks where a spec might be enough. The speaker emphasizes that in research, knowing exactly what's happening under the hood is paramount, especially given the complexity and rapid evolution of ML models.
17. The rate of consequential big idea generation in ML might be constant, but the standards for experimental rigor and overall quality have increased significantly with more researchers. This is a counter-intuitive but important observation. Despite a massive increase in the number of ML researchers, the generation of truly groundbreaking "big ideas" might not have accelerated proportionally. However, the *quality* and *rigor* of the research have dramatically improved. Papers now undergo much more thorough experimental validation, baseline comparisons, and sophisticated mathematical analysis compared to decades past. So, while the number of *ideas* might be constant, the *impact* and *reliability* of the research output have likely increased.
18. Internal company research reporting can be more accurate in conclusions but less thorough in write-ups and baseline comparisons than academic publishing. There's a trade-off between industry and academia in how research is documented. Big AI labs excel at drawing accurate conclusions from experiments because the stakes (real-world consequences) are higher than just getting a paper published. However, internal reports are often less detailed than academic papers. While academic papers might have questionable baselines, the best academic work is very thorough. Internal research might skip some baselines or detailed write-ups because the primary incentive is often shipping shippable improvements, not necessarily exhaustive scientific documentation.
19. The influx of researchers has led to higher bars for entry and a greater emphasis on engineering skills over pure research taste and exploratory abilities. The ML field has become much more professionalized. The sheer volume of people trying to get into AI means the bar for entry is higher. Furthermore, the focus has shifted. With scaling driving so many improvements and abundant infrastructure to build upon, strong software engineering skills are now often more critical than just raw research taste or the ability to do purely exploratory work. Building on existing codebases and integrating with complex systems favors those with robust engineering backgrounds.
20. The future of RL research will likely see ideas cycling back into fashion, with offline RL and sim-to-real techniques (like in robotics) being particularly interesting. RL is a field where ideas ebb and flow. Concepts that were too early or didn't quite work might resurface and find success later. Offline RL is seen as a promising area, and the concept of "sim-to-real" (training in simulation and transferring to the real world), which has been effective in robotics, is also relevant to LLMs. The speaker expects a return to learning from real-world deployment data, suggesting a continued blend of simulation and real-world experience.
21. Coordination between leading AI labs on safety is plausible due to common viewpoints, but personal rivalries could be a hurdle. While competition is fierce, there's a foundation for collaboration on critical issues like AI safety. The leading AI labs share a reasonable amount of common ground in their vision and have even collaborated recently on safety initiatives. However, the speaker acknowledges that personal "bad blood" between key individuals could complicate these efforts. Ultimately, if the need for coordination becomes clear, it's likely to happen, but it won't be without its interpersonal challenges.
22. AGI timelines are likely underestimated; engineers and researchers tend to be overly optimistic, and historical analogies like self-driving cars suggest longer development cycles. This is a solid critique of many AGI predictions. The speaker agrees that there's a consistent bias to underestimate project timelines, often by a factor of 2-3x. Applying this heuristic to AGI suggests it will take longer than many predict. The analogy to self-driving cars, which have taken much longer to reach full autonomy than initially anticipated, further supports this. However, the counterpoint is the AI accelerating its own development, which could defy intuition and shorten timelines, making predictions highly uncertain.
23. Tinker is a low-level fine-tuning API from Thinking Machines that abstracts away GPU and distributed systems complexities, aimed at sophisticated ML practitioners. Tinker is a new offering designed to simplify a complex part of the ML pipeline. It provides a set of low-level primitives for training and sampling, allowing users to express various post-training algorithms without getting bogged down in the nitty-gritty of GPU management or distributed systems. It's positioned as a service, similar to how OpenAI and Anthropic offer sampling APIs, but for training. Currently, it's best suited for those with a deep understanding of ML who want to leverage these primitives.
24. Tinker's ambition is for future AI companies to build on its infrastructure rather than developing their own, eventually becoming more user-friendly for a wider audience. The long-term vision for Tinker is significant. The hope is that it becomes the foundational infrastructure for many new AI companies, allowing them to focus on building sophisticated custom models without reinventing the wheel on training infrastructure. Over time, Thinking Machines plans to make Tinker more accessible, adding higher-level components and tooling so that users won't need to be experts to use it, enabling them to simply define their business problem or model spec.
25. Thinking Machines plans to release new models and enhance Tinker with multimodal capabilities and larger job scaling in the near future. Looking ahead, Thinking Machines has concrete plans. They'll be releasing their own models, and Tinker will see continuous improvement. Key enhancements include adding multimodal functionality (handling various types of input and output) and significantly scaling up the size of jobs that can be processed. This signals a commitment to both advancing their own AI research and providing robust tools for the broader community.
🎯 Expert Opinion
This transcript offers a fantastic look under the hood of AI development, touching on everything from the historical evolution of OpenAI to the nitty-gritty of research management and future directions in RL. From an expert perspective, several themes stand out as particularly significant and warrant deeper analysis:
The "Hindsight is 20/20" Phenomenon in AI Development: The idea that ChatGPT-level AI could have been built much earlier with full hindsight is a powerful illustration of how much we learn through trial and error. It underscores that current AI progress isn't just about raw compute or data, but also about the accumulation of knowledge regarding architectures, training methodologies, and alignment techniques. This has huge implications for how we think about future breakthroughs. If we can learn to better capture and disseminate this "hindsight," we might accelerate progress even further. However, it also suggests that predicting *when* the next major paradigm shift will occur is inherently difficult, as it relies on emergent understanding rather than predictable engineering timelines.
The Evolution of Research Organizations: The shift from an "academic, ragtag" group at early OpenAI to a more structured, engineering-focused organization is a common trajectory for successful tech ventures. This highlights the tension between pure exploration and goal-oriented execution. While the former is crucial for novel discoveries, the latter is necessary for scaling and productization. The success of projects like Dota demonstrates that combining these approaches, with significant engineering investment, is key. The challenge for any growing AI lab is to maintain a culture that fosters both serendipitous discovery and disciplined execution, a balance that seems to be a constant negotiation.
The "Dead Ends" as Foundational Investments: The discussion of projects like "Universe" and robotics as "dead ends" but also as valuable capacity-building exercises is a critical insight into R&D. These weren't failures in the traditional sense; they were investments in infrastructure, talent, and learning. The ability to undertake ambitious, high-risk projects that don't immediately yield productizable results is a hallmark of leading research institutions. It suggests that evaluating the success of an AI lab shouldn't solely be based on immediate product launches but also on its capacity to train researchers, build complex systems, and explore the frontiers of knowledge, even if the direct path to a breakthrough isn't immediately apparent.
The Nuances of Research Management and Culture: The differing management styles and the emphasis on building the "right culture" early on are vital. In a field as dynamic and intellectually demanding as AI, effective leadership isn't one-size-fits-all. The recognition that different teams and project phases require different management approaches is sophisticated. Furthermore, the idea that culture is hard to build later implies that foundational values and working norms established early on have a long-lasting impact. This is especially relevant as AI companies scale rapidly; maintaining a healthy and productive research culture becomes a significant leadership challenge.
The Shifting Landscape of Skills and the Rise of Engineering: The observation that engineering skills are becoming more important than pure research taste is a significant trend. As AI systems become more complex and built upon existing codebases and infrastructure, the ability to engineer robust, scalable solutions is paramount. This doesn't diminish the importance of research ideas, but it means that translating those ideas into tangible, impactful systems requires a strong engineering foundation. This shift will likely continue to shape hiring practices and educational priorities in AI.
The Cyclical Nature of RL Ideas and the Promise of Offline RL: The cyclical nature of ideas in RL is a well-known phenomenon. Concepts that fall out of favor often return with new theoretical insights or computational capabilities. The emphasis on offline RL is particularly interesting. It represents a move towards more data-efficient and potentially safer learning paradigms, as it allows training on pre-collected datasets without requiring constant interaction with an environment. This is crucial for domains where real-world interaction is expensive or risky, and it aligns well with the vast amounts of logged data available in many AI applications.
The Underestimation of AGI Timelines and the Self-Acceleration Feedback Loop: The critique of AGI timelines being consistently underestimated is a well-founded observation. The analogy to self-driving cars is apt, as complex, real-world systems often present unforeseen challenges. However, the counter-argument – the AI accelerating its own development – is perhaps the most compelling and uncertain factor. This positive feedback loop could indeed compress timelines in ways that are difficult to predict. It suggests that while engineering bottlenecks might be real, the rate of algorithmic and conceptual breakthroughs, potentially driven by AI itself, could be the dominant factor. This makes forecasting AGI a truly complex probabilistic exercise.
The Strategic Importance of Infrastructure Abstraction (Tinker): The development of Tinker by Thinking Machines is a smart strategic move. By abstracting away low-level complexities like GPU management and distributed systems, they are creating a foundational layer that can accelerate development for a wide range of ML practitioners. This is akin to how cloud computing democratized access to powerful hardware. As AI models become more sophisticated and require more specialized training techniques, tools like Tinker that simplify the process of fine-tuning and custom model development will become increasingly valuable. Their ambition to make it more user-friendly over time suggests a long-term vision to broaden access to advanced AI capabilities.
The Interplay Between Academic and Industrial Research: The comparison between academic publishing and internal company reporting highlights the different incentives and strengths of each. While academia often excels in thoroughness and detailed write-ups, industry labs are often better at drawing accurate conclusions due to the direct impact of their work. The desire to improve research writing culture within companies and potentially bridge the gap between these worlds is a critical area for advancement. The ideal scenario would involve the rigor and detail of academic papers combined with the accuracy and real-world grounding of industrial R&D.
The Evolving Definition of "Talent" in AI: The shift towards engineering skills reflects the maturation of the field. While creativity and deep theoretical understanding remain vital, the ability to implement, scale, and integrate complex AI systems is becoming equally, if not more, important. This suggests that future AI talent will need a more interdisciplinary skillset, bridging the gap between theoretical research and practical engineering. The "weirdness" of early AI researchers might be replaced by a more structured, problem-solving orientation, driven by the need to build and deploy increasingly sophisticated AI.
Overall, the transcript provides a rich tapestry of insights into the past, present, and future of AI research and development. It emphasizes the iterative nature of progress, the importance of infrastructure, the evolving skillsets required, and the inherent uncertainties in predicting future breakthroughs.
Kanal: Cursor