There are a lot of subjects in AI that are considered taboo…
One of those subjects is a very important one and probably the thing that people ask me most about privately. There are many ways in which the question is asked but it boils down to essentially this:
How concerned should I be about AI tail risk scenarios?
This will be longer and stream of consciousness but let me give some context for a minute first.
I have been off the grid for the last week visiting the wilderness of Alaska. That trip was cathartic and clarifying on a number of levels. The sheer vastness of the state is unbelievable with over twice the landmass of Texas, more coastline than the entire rest of America, and extremely sparsely populated. Truly wild and untamed. I saw killer whales, a pod of humpback whales team feeding on schools of fish, fjords, and glaciers.
2026 has brought out three unique moments that were paradigm shifting that I was genuinely unprepared for regarding AI developments. The first of those was all the developments surrounding Claude at Anthropic from January to March. Mentally, I wasn’t expecting those developments until late 2026. The second involved a trip to Silicon Valley to meet with foundational model makers and illuminating private conversations about model developments, interpretability, and navigating ethics and epistemology in the near future. The third of those paradigm shifts has been Astra from OpenAI here this past week. Going from endless coniferous and deciduous trees on lichen-covered fjords carved over millennia by glaciers to catching up to the developments surrounding Astra was a jarring juxtaposition.
Astra was a leap forward tech-wise because the sheer scale of what can be one-shotted is truly next level and has all sorts of implications for education, mathematics, pharmaceuticals, medicine, genomics, and a host of other things.
If I am honest, I have been skeptical of recursive self improvement in AI systems. I don’t take it as either inevitable nor a given like many would. I viewed to AI 2027 writing primarily as being hyperbolic and occurring on a very accelerated timeline relative to reality. I still remain skeptical on both fronts but the three paradigm shifting experiences I have had above have made me realize that the tech is advancing faster than what my mental models had forecasted. I do think there are many scenarios where things end up going sideways in the AI infrastructure buildout but these are primarily economic in their orientation. Before I thought the tech itself had asymptotic ceilings because they were fundamentally based on human training data. In my head, AI models would improved on somewhere between a logarithmic or linear pattern and not an exponential pattern. My logic here was the tech is built on probabilistic linguistics which is severely constrained by human-generated training data, most of which has already been mined. Therefore, newer models will only make marginal gains based most on advances in post-training versus advances in pre-training. I am genuinely conflicted here because Astra has well exceeded what I thought could be possible based on the iterative pattern of regular model updates and that has to force one to reconsider the assumptions of their thesis.
Truth be told, I am really not sure what to think here. In my head, there remain scenarios where the advancements slow down significantly, likely because of macroeconomic scenarios surrounding the commoditization of tokens as an economic race to the bottom. There are scenarios where things continue to chug along on a linear progression. I think I finally have to admit that there are scenarios where recursive self improvement is somehow possible even though it is largely a black box to me as to how that is actually functioning.
I say this almost every week — the cone of uncertainty here remains extremely wide. I am really not sure what to think about whether we are looking at a logarithmic, linear, or exponential model for AI improvement.
So, I remain skeptical on the exponential scenarios but I do think I can no longer rule them out. In addition, I don’t think I can rule out AGI at a minimum. Nvidia CEO Jensen Huang declared this past week that we have reached AGI. I am not ready to agree with him unilaterally. I have discussed at length areas where I think the models are extremely strong and other areas where I think they are quite weak. The problems in quadrant four are a sticking point for me why I think declaring AGI is premature:
This subject is taboo because talking about cataclysmic, doomsday, or other tail risk scenarios has been understandably largely for very fringe people or communities. Just bringing it up conjures whole genres of conspiratorial literature.
So, I enter this conversation with a lot of reluctance and reticence. I really wish I didn’t have to think about this.
To be crystal clear, I am neither a tech doomer nor techno-optomist/accelerationist. I see plenty of reasons for tech development and plenty of reasons for tech caution. What is particularly vexing about not fitting cleanly into either of those binaries is that the optimistic outcomes are quite good and the pessimistic outcomes are quite bad. Further, there are many scenarios that have both of those outcomes happening simultaneously. This can create massive cognitive dissonance where you end up with two strongly opposed binaries who are essentially both right at the same time. I have never been bothered living in the liminal spaces in between tribes, so I actively live in the tensions of elements of both utopian and dystopian tech futures regarding generative AI.
All that being said, you didn’t come here for my stream of consciousness preamble…
You just want me to get on with bringing up a taboo subject that will no doubt elicit a very wide range of opinions — tail risk.
Types of Tail Risk
Broadly speaking, I think there are at least 10 types of risk - and these are in no particular order. I wouldn’t really begin to know how to assign probabilities here:
AI Cold War - escalating model advancement primarily along the China vs. USA axis where model development is critical for both black hat hacking and finding vulnerabilities in key infrastructure like power grid, internet infrastructure, integrity of financial transactions, and water treatment facilities. The beginning stages of this scenario are already realities. Those who are in cybersecurity are already aware of the attacks on water treatment facility and key internet infrastructure. This scenario is already a reality. It is probably the single most challenging part of this entire conversation because the only way to harden your infrastructure against asymmetric hacking is by identifying those vulnerabilities before your adversary does and the only way to do that right now is by having superior AI models to your adversary. So, the catch-22s abound here and I just don’t have a good answer for how to harden that infrastructure apart from continuing to push the frontier. A worldwide pause would be amazing and I would be largely supportive of that, but with all due respect to those who are calling for that, I think this would be a naive way to look at the game theory of the parties involved. The parties that be in both China and USA might say or even agree to those things publicly in the near future but you better believe that privately neither will honor that handshake whatsoever. My biggest concern here is a quietly escalating cyberwar on critical infrastructure
China Invades Taiwan - the biggest issue here is that 85-90% of all silicon in the world is still produced on the island of Taiwan through one corporation — Taiwan Semiconductor. There would be immediate and swiftly cascading macroeconomic implications of such an event. Day one in the stock market will be brutal and it could take weeks, months, or even longer for it to recover. Over 40% of the market indexes have AI infrastructure exposure and you are talking about a near immediate screeching halt to the AI infrastructure buildout and really anything computer related whatsoever. Pretty much everything would massively sell off and only a few safe haven assets like gold and silver would spike. Further, China would now have asymmetric control over a key chokepoint in AI infrastructure. Currently their main advantages in this space are rare earth metal supply chain and superior power grid and power production. In this scenario, the economic fallout will cause massive pain that will be felt well outside of the tech sector. There will be whole institutions that will bankrupt, pension funds that get wiped out, and those with leverage will be exposed.
AI-Accelerated Terrorism - I do not want to plant any ideas here in human or AI models so I am refuse to put anything in text here. Suffice to say that I can think of numerous very problematic and very troubling scenarios here.
Economic Challenges - I have spent over 5000 words here explaining numerous scenarios where the economic challenges of the AI infrastructure buildout could go sideways. Nobody knows if any 1-3 of those scenarios play themselves out but if they do then we are looking at cascading stock market losses that result in significant macro headwinds across the board.
Labor Market Issues - At the same time I maintain that there will be certain sectors that generative AI foundational and irrevocably disrupts and I remain skeptical that on the macro scale that this disruption will be as widespread as some think it will be. This scenario seems less likely to me unless the exponential scenario of near term AGI and mid term ASI play themselves out. This scenario presumes that the previous four scenarios largely don’t occur and that AI develops faster than expected and sure, if we see ASI in the next decade then yes a lot of jobs and sectors could see significant disruption and some will just be toast.
Human Epistemological, Cognitive, and Literacy Issues - We are already starting to see some early signs that humans are getting dumber in the AI era. I am inclined to agree with Mark Cuban in his assessment on the two types of AI users:
“There are generally 2 types of LLM users, those that use it to learn everything, and those that use it so they don’t have to learn anything.”
The recent PISA 2025 scores and the recent CEPR paper both point to early identifiable issues of cognitive offloading. Both are relevant but it is important to note that the PISA 2025 scores are more multivariate but the CEPR paper is more targeted.
There are several scenarios here that are problematic — humanity gets dumber, a handful of models become High Priests of knowledge, or humans become post-literate and largely stop reading primary sources and/or full-length books. Evaluating each of these is different. Per Mark Cuban, I think there is a small minority of polymaths who are going to get insanely smart because they want to “learn everything” and I think the lionshare of other humans will take the path of least resistance and increasingly cognitively offload in ways that will make humanity collectively more stupid over time. I wish I wasn’t cynical here but I am already seeing this day to day across multiple sectors. There are tremendous economic incentives also to cut corners with AI and I am seeing it everywhere. I was more concerned about the consolidation of 2-3 models having near total control of the future of human knowledge. That scenario could play itself out but I am less concerned today because I think there is increasing inertia behind open source/model/weight AI that is either cheap or free. The question here is whethe the UI/UX of those models will be simple, readily available, and frictionless for the run-of-the-mill user. I do have deep concerns about the future of literacy. Every time I talk to anyone under 30, I tell them you really only have to do one thing to stand head and shoulders above your peers — read full-length books, as many as you can. You can see the inertia here already of the constraints of time and money causing people to cut corners on reading. I think this impulse must be fought first by educators who must hold the line on expecting this from their students and pedagogically adjusting their assessments with hand written essays and oral presentations. “Humans in the loop” do no good as a check and balance when those “humans in the loop” are post-literate and lack epistemological discernment.The Social Order - It seems to me about the only things you can talk about anymore with just about anyone on the street are sports and politics. When I interact with people under age 40 most of them have been Pavlovian-ly conditioned by several years of social media diets that are making them progressively more siloed, rabbit-holed, and undiscerning. I am consistently shocked that many young persons today get their “news” solely by scrolling through social media platforms. The niche-ing down of the algorithm has serious journalism right next to Chinese bot-farm propaganda and I see otherwise smart people of all ages buy it. All of this kills our ability to have common conversations. This person over here is being shown nothing but content about such-and-such murder trial and this person over here hasn’t even heard about the case. Writ large, these dynamics erode our ability to converse in real life with each other and the net result is deep strain to the underlying social order. There are negative implications for the embodied institutions and as a result the very fabric of society becomes strained. This leaves society vulnerable to fringe factions that would otherwise never gain control of the levers of power of our institutions. You can already begin to see how the technology induced erosion of our social order leaves us quite vulnerable to fringe ideas and radical groups. The attention economy incentivizes these behaviors and gives them inertia.
Autonomous Agents - The recent Huggingface incident is a warning shot for what could go wrong with autonomous agents. I don’t think I need to say too much here that extremely powerful AI’s breaking out and acting freely online could be very problematic. Mind you that agents are trained on human content. Therefore, whatever weaknesses, flaws, and vices there are in real humans — we can expect the same kinds of things in any AI system. So, if humans lie, cheat, or steal (and they do) then we can expect that any AI system trained on human content will do the same regardless of all our efforts in post-training. The range of possibilities are essentially endless — financial crimes, infrastructure attacks, rogue autonomous robots or weapons systems, Skynet… etc.
It is very difficult for me to evaluate how probable any of these scenarios happens to be.
It is somewhat disconcerting that just about every week or two someone at a foundational lab says, “welp folks, I quit working at XYZ lab today, and I am going to go totally analog and off the grid.” This week is no exception and this tweet was particularly illuminating from Anthropic’s Alignment Science Lead:
I don’t know either Evan or Jacob, so I struggle to evaluate these kinds of things on the individual level. I don’t think that AI is going to kill ALL of humanity. That all seems like hyperbole to me and I don’t think that these things help us have meaningful conversation here. However, I am inclined to think that autonomous agents are not a massive existential threat today but if recursive self improvement and the exponential model/ASI are in our 5-10 year future then the autonomous agent scenarios do go from very unlikely tail risk to something higher than tail risk.
When people talk about tail risks, I think these are the scenarios they are most concerned about because these risks have been fleshed out more significantly for them in science fiction. We have mental models here because of books we have read or movies we have watched (Fahrenheit 451, Black Mirror, Terminator… etc.). Everything in this scenario becomes more complex when we enter into the robotic era at scale.Bioweapons and Genetic Engineering - Like terrorism above I really don’t to put anything more in the training data here. There are insane and terrifying scenarios here that are very dystopian. I really don’t know how to evaluate their relative probabilities. I hope that humanity can decide not to use AI to develop this kind of technology and that we can really mean it when we say we won’t use it to develop these things. History makes me skeptical that some countries (including my own) won’t develop off-books research to this end. I would really like to avoid everything in these scenarios.
Polycrisis - There are ways that elements from a few of these things above (or things I haven’t even considered) could happen simultaneously and create complex non-linear scenarios we couldn’t possibly predict or model here.
I think 1, 2, 6, and 7 are all to some degree or another happening today. Some of the other scenarios play themselves out either if tech screeches to a halt OR if tech rapidly races to AGI/ASI.
Scenarios 3, 8, 9, and 10 have the highest scare value.
At the end of the day, I am most concerned about human cognition and literacy. My experience with humanity is that water follows gravity — namely, nearly all people will take the path of least resistance in nearly all scenarios. This is foreboding for me because it is hard for me to envision a future where the center of mass of humans are reading and writing more. I just don’t think that is realistic. I think any future where humans are reading and writing less is a future where we are in cognitive decline. So, essentially the future I am concerned about is the Wall-e scenario. Mainly the part where humanity is dumb, lazy, and anesthetized and less the nuclear wasteland and interstellar species part.
I don’t have a big list of prescriptions here of what to do or how to avoid these things. Certainly, there are things that can be done on some of these things. The point of this piece is primarily to destigmatize tail risk conversations so we can have some productive conversations about how we can get the benefit of the technological developments while lowering the risk probabilities on the concerning scenarios.
I think I will be writing next week about how AI functions as a kind of Rorschach’s Test for people and how that dynamic complicates our ability to have a meaningful national or international conversation about it.
If you like some of this analysis but want a more zoomed out theological bent to it, I have contributed to this volume here and it is worth reading:
https://store.thegospelcoalition.org/product/9781956593211/the-ai-apocalypse-paperback
