Intellectual Progress in 2025

It is now 2026 and we are half way through the decade of the 2020s. If we think back to the halcyon days of January 2020 certainly a lot has happened, especially in AI¹. The first half of the decade has essentially been the discovery and then incredible exploitation of scaling. We have scaled the compute and data poured into these models by many orders of magnitude, ‘racing through the OOMs’. As we come into 2026 there are a few OOMs still left to go but the end here is clearly in sight. It has become fairly clear that pretraining on webtext alone does not lead to AGI, which was the common belief in 2022. Now the question is whether scaling RL across all environments will. In some sense, it must insofar as we explicitly train our model to solve all the environments and tasks we care about for AGI. In another sense the open-endedness and self-bootstrapping creativity is still missing. We can certainly consider the AI companies as a whole as a kind of proto-AGI where any specific task, once made into a benchmark, can be solved in short order and they have internal processes to decide what tasks to make benchmarks of, but ultimately for an AGI we want the models themselves to be making this determination. Maybe this process will solve itself simply by meta-learning on a sufficiently large corpus of tasks, maybe it will not. That will be the crucial question for the next few years.

Before we go directly to 2025 alone, it is also worth stepping back and considering a slightly more zoomed out view. Pretty much everything I have achieved, I have done in the 2020s. In Jan 2020, I had only a few papers out, essentially no citations, and was entering the final year of my PhD. Since then I have,

• Finished the PhD with many papers published and an actual strong academic record thanks to my fellowship at Sussex and reached the frontier of understanding and contributing to predictive coding and active inference
• Started blogging and eventually got this blog to some small degree of readership
• Went through a postdoc at Oxford and finally understood the landscape of biologically plausible approximations to backprop, single shot associative memory models, and information-theoretic exploration objectives.
• Joined Conjecture, got into interpretability, and helped start the SAE boom
• Fleshed out my thoughts on alignment, what human values are, the shape of the singularity, and learning theory
• Helped cofound Apollo at the beginning
• Moved to the Bay Area and founded Zyphra where we are slowly building the resources and team to start understanding the real deep questions in intelligence and alignment.

Obviously looking back on it there are many things I could/should have done better but at the same time it could also have gone a whole lot worse. So that is important to keep in mind. The future is uncertain and who knows how things will go but the key thing is adaptability, flexibility, and just circling around and coming back to the same few core interests and questions again and again with different perspectives. Although 5 years is not long in an absolute sense, it is long enough to begin to see the paramount importance of adaptability and maintaining plasticity in learning and interacting with the world, and to have made predictions and been able to assess how they have gone. At these timescales the derivatives and second derivatives begin to take a serious hold of the process in a way that is not visible in the day to day.

One thing that has been interesting to me is seeing how I have tried to navigate the academic gravity turn myself in my own life. Another thing that has become super apparent over time is just how much output compounds often in weird and serendipitous ways. Some miscellaneous paper or blog post or other activity has led to interesting connections, insights, or new opportunities at a far higher rate than I would have naively predicted, and these things build on themselves over time. Academic citations also compound themselves over time it appears. This year was perhaps one of my worst in terms of legible paper outputs because of my focus internally at Zyphra and yet this year my past papers got more citations than ever before.

One thing I have noticed about myself intellectually is that my main contributions and interests are in understanding, distilling, and then trying to unify existing fields rather than striking out and exploring totally novel vistas. My primary strength seems to be as a synthesizer, which is also something I do via my blogging. Many of my best² papers involve taking some complex and scattered seeming field and showing how everything is actually incredibly simple, even trivial when written out and understood in a certain way. This is an interesting inclination but at the same time it is not really sufficient for some of the progress I want to make which involves actually trying to directly explore and attack some of the fundamental questions of intelligence. That is something I have to work on in the coming years.

Coming back to 2025, one interesting thing I’ve noticed, at least for me and on my social media feeds, is that much of the excitement and frenetic energy of 2024 has drained away, especially on AI twitter. There are still big model releases but these are much more humdrum events and when we see a new model gaining 3 points on the competitors on eval X we all kinda shrug and go ‘that’s nice’ vs ‘everything is going to change!’. The most surprising thing was how GPT5 was such an absolute nothingburger compared to GPT4 or chatGPT. The main dialogue point about GPT5 was how the presentation included many crimes against graphs and the field of visual design in general. Similarly Gemini3 and Opus4.5 made a few small waves but nothing anywhere major. A massive feature of the memetic landscape of 2024 (only 1 year ago!) was the increasingly unhinged rumours and vagueposting about ‘Strawberry’ which of course turned out to be CoT RL³. The days of vagueposting and secret lab advances (teased in public) seem to be over. There might well be secret lab advances nowadays but if there are they are keeping much tighter lids on them than previously. In some sense it feels like the AI field has calmed down in 2025 compared to 2024 after the initial deepseek hype. Progress has been steady all the same, and in some sense has been faster than 2024 in that everybody is reaping the low hanging-fruit of scaling CoT RL (which probably has at least another year till exhausted), but each advance is treated much less rapturously. I think some of this is that in the 2022-2024 era LLMs were primarily traversing the median human range where each improvement felt like a massive advance in capability and usability. Now the LLMs are entering the superhuman range for the tasks they are good at which means the average person feels little difference and the vast majority of tasks do not require the ‘full intelligence’ of the AI. If you talk to somebody who won an IMO silver vs an IMO gold, can you tell the difference?

In my mind, this is creating a vibe-shift towards longer timelines that is only slightly less unwarranted than the breathless ‘AGI tomorrow’ fever of 2023-2024. Realistically, frontier LLMs are already strongly superhuman at basically all tasks requiring crystallized knowledge and pattern-matching reasoning, and they are increasingly getting better at fluid intelligence via long CoT reasoning. We are only a small number of breakthroughs away from AGI. When these breakthroughs happen is hard to predict (it could be tomorrow!, or a few months ago in secret, or in 5 years), but once they happen AGI will be here much sooner than we will expect. All the infrastructure for both training and inference is primed and ready. The data-centre buildout will likely reach its climax in the next 2-5 years, but my opinion is that the scale we have right now is sufficient for AGI and probably has been for the last few years. There is thus a massive compute and infrastructure overhang being created that will allow AGI to scale incredibly rapidly once created, at least for a while. Industry trends though are relatively closely (although not entirely) vibe-following and hence we may see a cooling of interest and excitement about AI at ironically precisely the wrong time.

Broadly, this is because society is slowly assimilating LLMs into its worldview and understanding. This is going to cause some waves and change a lot of jobs but clearly not going to be utterly revolutionary in the way AGI would be. It will be e.g. like electrification or some large 19th century industrial revolution technology but not the creation of a god. People who are used to LLMs then forget the original promise and dream of AGI. But the AGI is still lurking there in the future, not so distant, and coming closer.

For me, personally, I think this year has been an important one in understanding where we are at and what remains for AGI. The path is a lot clearer to me now. The same is true of alignment and this is somewhere where I need to seriously take some time and write up my thoughts. My hope is that I will be able to get to this in 2026. Things are still advancing very rapidly though so we will see to what extent they will already get obsoleted and need revisions due to events.

Organizationally, things at Zyphra have been interesting for me as Zyphra has scaled further. We’ve now grown to about 50 people at which size we are beginning to get a bit unwieldy to run entirely on personal relationships. For instance there are starting to be people in the company that I only interact with perhaps a few times a week instead of everybody every day. We are still at the size where everybody knows everybody but I can easily start to see how after another doubling or so this will start to become increasingly infeasible as well. It’s going to be interesting figuring out how to manage and lead increasingly large teams as Zyphra continues to scale as well as dealing with the additional coordination costs and inefficiencies that this brings. I talk a good game about coordination costs and scaling etc in various posts in theory – it’s also fun to see how this stuff works in practice⁴.

In any case, management is a game I’m becoming somewhat experienced at although, like most things, raw talent trumps experience very quickly (which becomes increasingly worrisome as I get older and fluid intelligence is replaced by crystallized experience)⁵. I don’t yet notice any declines in mental acuity (nor should I), however I have noticed my personality mellowing out slightly from the intense neuroticism of my early 20s, which is interesting.

One interesting thing that Zyphra affords me is that because we do so many different things, I get almost a synoptic view of AI progress across modalities and between research and engineering. This is unlike almost everybody else except perhaps higher ups at labs who only get a much more partial view of the specific problems they are working on. Hopefully I can use this interesting vantage point to formulate some interesting ideas. On the other hand, it has the disadvantage of further removing me from the day to day work on e.g. interpretability or learning algorithms that fueled many of my earlier ideas. Reading papers is only a partial substitute. Ascend high enough and you can see nothing but the clouds, which is why, I think, beyond simple tails-come-apart effects and strong incentives, that most highly placed CEOs, politicians, etc always end up spouting such unoriginal platitudes despite the theoretically fantastic vantage points they occupy and being surrounded by some of the smartest and original thinkers around⁶.

While Zyphra’s output has been pretty lacklustre this year, we’ve made some very interesting internal progress which hopefully will start to resolve itself next year. Perhaps some of the most interesting things, at least to me, have been conceptual deconfusion about some key steps on the path to AGI. Every year things become clearer and clearer and generally I am so happy to be alive at this critical point of human history and in the middle of one of the greatest intellectual discoveries humanity will ever make.

My work at Zyphra has also brought me into more contact with higher-ups at large companies, investors, billionaires etc, and I feel like I understand this world a fair bit better now than I did last year and much more than I did before Zyphra. Most of this is fairly intuitive and obvious once you think about it. Billionaires and famous people are fundamentally still human and face many of the same constraints and make the same mistakes that regular people do. As far as I can tell, nothing special happens at increasingly higher levels; it’s just regular people doing regular people things. As many others have observed, there is no special secret room where all the super competent people secretly run the world. Of course many/most of the people at this level are highly competent but in regular person ways.

One other tangential but interesting point I have realized is how differently we run Zyphra and our strategy is compared to the regular YC startup playbook. Going into Zyphra I, of course, read heavily into the Paul Graham and general YC mindset and culture and I know many people who have gone through YC and imbibed its general cultural mores. This includes stuff like ship an MVP as quickly as possible, growth is the solution to all problems, find customers quickly and iterate, customer focus and obsession, etc etc. Zyphra has basically done none of this and has nevertheless been able to raise and increasingly expand by leveraging promises of AI research, and strategic relationships with large companies based on hardware and shared technical milestones which are extremely difficult from the standard YC-verse. Interestingly Sam Altman, despite being head of YC (!), followed a similar strategy also diametrically opposed to YC dogma with OpenAI. AI companies, at least for now, seem to play by very different rules than YC SAAS companies, and that is super interesting to see and understand in practice.

This is one of the underrated things I like about Zyphra, that none of the cofounders of Zyphra were large existing lab alums or were famous in any way, and yet somehow we have managed to build our own little AI lab and get it to the point where we can start competing with the big players in the field. I don’t know of any other AI lab that has accomplished this, except perhaps Magic.dev.

Another thing I have realized about startups is that relatively low valued acquisitions are pretty common and this actually substantially derisks startups compared to employment even more. I.e. at Zyphra we received a low 8 figure acquisition offer after only 1.5 years, which we turned down, but which would have set me and the other cofounders up financially for life⁷. Now, we are at a higher valuation still but ignoring that, we have seen this happen to many other companies as well and there is seemingly, at least in the current market, substantial value in simply building a team and demonstrating technical capabilities which is fairly easy to do and has the potential to result in rapid mid-range financial outcomes if done correctly. I never really realized this before and thought that startups were much more binary in terms of massive unicorn success or absolute failure but in fact there seems to be, at least in AI, a substantial ‘safety net’ of being acquired by some big company for your team as an outcome which while not propelling you into the ranks of the super rich will net you out a comfortable life and a decent big-company job. Of course, like all things, this has to be finessed correctly and you do have to demonstrate value somewhere.

Perhaps the most interesting area of progress this year for me was my blog. Not only did I get a fair few posts out, I managed to flesh out in some more detail my general theories of alignment and how society evolves in the post-AGI world a lot more than I have in the past, and hopefully there will be some more coming in that direction in 2026. This year feels about as productive as 2023 for me in terms of blogging. While I think I still originated most of my core ideas in the 2022-2023 period especially at Conjecture, I feel that I deepened and better understood them during the previous two years and especially this year as we draw somewhat closer to AGI and the dust begins to settle over this phase of the AI game.

Generally I’ve been surprised by the positive reception my blogging has received, with some fairly big-name people having read it and appreciated it. This surprises me mostly because nothing I say is that original or well thought through. Most of it is vague musings and unsupported ideas I have. I have been realizing that the only reason that this is working is because there is a surprising paucity of original intellectual contributions in the world that are accessible to those outside of academia. Lesswrong and the greater rationalsphere is not perfect but is a shining beacon of such creativity. I really know of nowhere else on the internet that comes close. There are certainly isolated blogs and substacks out there as well but they do not cohere as well as the rationalsphere. Everybody reads the same internet and knowledge is the real constraint. It’s always amazing to me in practice how small the world is. We are extremely extremely far from the EMH world of infinite agents pursuing the same thing. In practice the world is incredibly high dimensional and there is only a small fraction of the current population who have the capability, inclination, and prerequisites to engage at the frontier of anything. I suspect honestly this will be true for a long time even with AGI. The combinatorial high dimensional space of ideas can grow exponentially which even a FOOMing AGI population will manage to match but likely not exceed.

In terms of alignment, no super dramatic updates but a slight change of focus. It’s clear that my original thinking in 2022-2023 about LLM alignment was largely correct. The current paradigm of LLMs poses little threat and can be aligned relatively straightforwardly. The next era of RL agentic LLMs is slightly more tricky. Here though I am somewhat optimistic because the principle challenge to aligning today’s RL-ified LLMs is reward hacking which is also a fundamental bottleneck on their capabilities. I’ve had some ideas about reward hacking, but generally the alignment of incentives between capabilities and alignment bodes well here.

The next frontier of alignment for me is figuring out how to stabilize continually learning online agents which just go off and autonomously interact and learn from the world. This seems to me to be a significantly harder problem than aligning a static set of weights which can be thoroughly tested before deployment. This is true even if the ‘base’ methods of alignment e.g. RLHF/RLAIF, constitutional character training, monitoring, redteaming etc still ‘work’ and the architectures are still the same old transformers that we know and somewhat understand. I’ve had some preliminary thoughts about this but a lot more thought needs to go into this. Of course, like original alignment theory, it is very hard to say concrete and correct things about future systems which we do not yet understand how to build. Beyond this, there are then questions about how to attempt to align a highly ‘polytheistic’ AI world with various agents with different utility functions and worries about e.g. cultural drift or malthusian competition between such agents removing any initial alignment. These are another class of concerns I am actively thinking about and hope to have some posts soon upon.

In terms of extracurricular learning this has again been a bad year. I simply don’t have time between Zyphra, keeping up with AI papers, blogging, and the multitude of other obligations I have. The days of academia with one meeting a week with my supervisor and just endless free time to explore whatever I find intellectually interesting seem sadly over. I think I’ve just got to accept this and not beat myself up about it until something dramatically changes in my worklife such that my schedule becomes suddenly freed up. I think this is unlikely in the near future, however. Sometimes I think about taking e.g. a three month sabbatical just to finish up my backlog and try to learn some long-standing fields of interest but unfortunately this is largely impossible. I’ve reached the point where opportunity cost rules everything around me, and if I continue onwards these constraints are only likely to bind ever more tightly. I still haven’t figured a way out of this and honestly likely there is no escape. Such is life.

In my personal life I feel pretty settled into the Bay Area now, which is nice. Probably the main thing I should do differently is engage more directly with the large AI and rationalist community here which I have so far largely not done. I’ve never been a social butterfly but it is important to try and get a sense of the lay of the land rather than staying a hermit. To some extent this is due to the geographical (and cultural) separation of the peninsula/Palo-Alto scene from the SF AI scene and also the Berkeley rationalist scene. But also just due to my more introverted nature who does not seek out large social gatherings or parties which are not controlled by strict social norms.

In terms of other events this year I did two speaking engagements. First a TED talk in Miami which was fun but mostly just discussing AI and AGI at a popular-science-ish level, and secondly I presented at the post-AGI workshop in San Diego for NeurIPS. This one was really fun and a great excuse to be maximally speculative. This presentation gave me a real push to finally condense and write up my vaguely formed thoughts which became the latest series of posts – so this was good!

This year I also travelled a lot more than in prior years averaging roughly one trip per month, mostly for work. This is a lot of travel and was both fun but also tiring and took up a lot of time. It is certainly fun going around and seeing various places, especially in the US which I had not explored before but at the same time the logistics of travel make it very hard to be productive which I have always found requires long periods just thinking and working in the same place without too much interruption. One advantage travel does have though is that airplanes are fantastic places to try to write annoying blog posts you have never found time to get around too. However this only works if you mostly know what you want to say. Doing original deep thought with a packed travel schedule seems very difficult if not impossible. The plane-written blog posts are mostly reaping and not sowing. There is certainly some kind of general tradeoff, at least for me, between experiencing and contemplating. If your schedule is packed with travel and work and you get a huge amount of input, however properly digesting and processing that input requires time of contemplation with little input. Perhaps I am just rederiving the super basic idea that rest is important, even for productivity over the long haul, but this has been interesting for me to see play out over the past few years. I never had this problem during my PhD when I was just mostly sitting at home all day reading papers and thinking. But now there is a real tradeoff. The ideal schedule seems to be something like perhaps 3-6 months of serious high paced works, perhaps interspersed with small few-days blocks of contemplation then a big 2-4 week holiday/contemplation block at the end to really try to pull everything together. Insights also require times of serendipity where you can just explore without being pressed in with obligations and always happen to me at the weirdest times. I’m not sure how to optimize these further really. The Christmas/New Year’s break always forms one such contemplative block for me but there also needs to be a similar one in the summer.