Sadly, they now realise that it is likely that all white collar labour is going to get automated in the next 5 years. As a result, many of them are re-discovering the meme of “Escaping the Permanent Underclass” from first principles.

It starts from the observation, through Cursor Agents or Claude Code, that AI is good enough to understand and execute complex plans. Some minor extrapolation shows that given the current rate of progress, all white-collar labour is going to get automated soon. Let’s say in the next 5 years.

After that point, it is obvious their smarts and tech proclivities won’t matter and that they are going to get out-competed by AIs. Plausibly, there will always be something that they can do, but there is a solid chance that they won’t, and they would like to not risk losing everything if this came to happen.

Their reasoning ends with the conclusion that they must amass enough capital or build enough income streams based on AI automation before this happens. Else, they will permanently become part of an underclass that has zero prospect of making big bucks, at the mercy of potential government redistribution programs.

—

Consider this very short article from Tyler Cowen, a famous economist who has been a bit skeptical about the prospects of AI.

If that describes you or someone you know, this article is for you. In it, I will explain that this line of reasoning misses how AI changes the landscape of politics and geopolitics.

Titles of property, shares and numbers on a ledger won’t protect anyone against swarms of autonomous killer drones. This is true regardless of whether said drones are controlled by private corporations, massive governments, or rogue AI systems.

In short, there is no escaping the permanent underclass, for the permanent underclass will include everyone who doesn’t have direct control over the most powerful AI armies.

Strategic Interests

Recently, a dispute occurred between the Department of War and Anthropic, over the use of AI systems as autonomous weapons and for mass surveillance purposes. Many commentators felt uneasy about the situation, and what it meant for freedom of contract and the like.

These commentators may not be ready for what is to come. This is the weakest these tensions will ever be. As time passes, AI systems only become more relevant to the strategic interests of governments.

As of 2026, Lethal Autonomous Weapons (LAWs, the high-brow name for “killer robots”) are being developed and used. Wikipedia has a long article dedicated to the race for LAWs, which it describes as:

A military artificial intelligence arms race is an economic and sometimes military competition between two or more states to develop and deploy advanced AI technologies and lethal autonomous weapons systems (LAWS). The goal is to gain a strategic or tactical advantage over rivals, similar to previous arms races involving nuclear or conventional military technologies.

Coincidentally, China has built the habit of parading larger and larger swarms of drones. For instance, Chinese entities hold many of the Guinness World Records entries for drones swarms. It started with 7.6k in September 2024 from Damoda (a Chinese drone swarm company), to 11k in July 2025, to 16k in October 2025 and to now 22.5k in February 2026.

—

While Think Tanks like the RAND Corporation have been writing for years about the interaction of AI and Cyber warfare, we now have concrete examples. In November 2025, Anthropic has publicly disclosed that China used Claude Code to orchestrate large-scale cyber attacks against western targets.

But this is not limited to virtual contexts. Beyond cyber warfare, AI is now used in “regular” warfare. Specifically, it is LLMs-based AI that is used, the same technology that powers ChatGPT and Claude. This is how the Financial Times introduces its article on the usage of AI in the US-Iran war:

AI is reshaping how the US military makes decisions in war — a shift clear in Iran, where the Pentagon says it struck more than 2,000 targets in just four days.

The unprecedented tempo of targeted attacks has been driven in part by AI systems that sift the torrents of intelligence data from drones, satellites and other sensors, generating strike options far faster than traditional human-led planning.

The conflict also marks the first battlefield use of “frontier” generative AI models, with AI tools widely used by civilians — from office workers to doctors and students — helping commanders interpret data, plan operations and provide real-time feedback during combat.

—

The above are existing threats, and they are already worrying. But with AI comes novel threats. While Weapons of Mass Destruction (WMDs) have historically required secretive programs, AI is democratising the development of chemical and biological weapons.

In a more prospective direction, AI-enabled technological development will help with the creation of novel weapons, that could provide a strong first-mover advantage to those building them, which RAND dubs “wonder weapons”. Such wonder weapons would be enough for the power that builds them first to promptly militarily dominate the rest of the world.

The most obvious wonder weapon is autonomous armies made of robots and drones. Any army that manages to automate away its need for soldiers, drone pilots, and even its generals, will hold a decisive advantage against armies that still depend on humans. Humans must sleep, but robots and AIs can track targets relentlessly. AIs can process information from hundreds of sources, quite faster already than humans: if you doubt it, try to out-speed an AI agent on a search query over the Internet or a long text document.

Less prospectively, AI can already be used for mass surveillance. The vision for mass surveillance that Orwell depicted in 1984 was unrealistic because of the scale of surveillance infrastructure it needed. From microphones and cameras in every home to the near constant manual monitoring of everyone, it was not tractable.

Nowadays, smart microphones cost less than 10$ and existing AI systems are already powerful enough to process that information at scale. Processing 2 hours of speech per day through AI systems would cost ~$285 per year.1 For comparison, in the US, the average yearly SNAP (food stamps) benefits amount is $2,244, and the average yearly retirement benefits amount is $24,852 (according to Social Security). AI-assisted surveillance is thus one to two orders of magnitude cheaper than these programs.

—

AI is already used for war and mass surveillance. It will only integrate itself more in these processes and become more strategic over time. For now, this is happening largely by automating old means, like with autonomous killer drones as well as AIs processing camera feeds and private correspondence.

However, a few actors are determined to build super-intelligences that can do more than that and create novel technologies, bringing war from autonomous killer drones to wonder weapons, and bringing mass surveillance from AI processing to dystopian BCIs that read people’s thoughts.

If, when thinking of AI, your mind goes to “AI will automate a lot of my trade”, you are severely underestimating the impact of AI. More importantly than automating generic white-collar labour, AI is being used to automate and amplify critical industries, regime stability (“domestic security”), and war.

In other words, understanding the Permanent Underclass as a separation between the haves and the have-nots is a very Marxist mistake, that applies to a technologically static world, and not one where AI is currently developed to entirely replace and exceed humans.

Severing the Interconnectedness of the Modern World

To some of my readers, the above will be enough. By re-contextualising AI in terms of its impacts on conquest and domination, they will naturally conclude that such considerations dwarf any discussion of a neo-proletariat in the AI era.

But at this point in the essay, most will still feel unsatisfied. Indeed, important questions remain unaddressed. For instance, if not between the AI capitalists and human labourers, what would be the sides in the conflicts downstream of AGI?

After all, world-altering technologies are nothing new. We built nukes in the middle of the 20th century, and somehow, democracies remained. Since 1945, entire business empires rose and fell, frontiers got redrawn.

In the end, property rights have always been backed by state violence, a medieval lord could always claim back the land rights of a peasant by virtue of having power.

If one claims that AGI will be the end of history, they must actually argue their case.

—

AGI completely changes the power dynamics at play in our world.

For now, in our current world, the fate of the world is largely decided through that of supra-human entities, like states, political parties or corporations. And for the better, these supra-human entities all strongly depend on humans for their power.

Without human labour, states cannot fill the ranks of their armies, cannot supply theirs with equipment, cannot maintain borders, and cannot keep defence systems operational. In other words: without human labour, states cannot defend themselves. To maintain their continued existence, states must manage to keep people alive and make productive use of them.

In our modern world, illiterate idle people are not very useful. States thus must ensure that their people are educated and motivated to some extent. Ambitious states – that do not merely survive, but actively expand their power – must also ensure that they have a large population of creative and driven intellectuals. Nurturing such a population takes more than basic survival and primary education.

Similarly, successful political parties must manage to appeal to voters; successful corporations must manage to appeal to their customers, employees, middle management and C-suite all at the same time, and so on. To the extent these organisations become more powerful, it is virtually only because they manage to gather the support of more people, whether it is voters, customers, employees or regulators.

That is to say, we live in a world where the worst mega-structures still depend on people and their approval.

—

There is a famous essay from 1958, entitled I, Pencil. It is fairly short, 5 pages, and I recommend reading it. But if you don’t feel like switching to another essay right now, here is its Wikipedia summary:

"I, Pencil" is written in the first person from the point of view of a pencil. The pencil details the complexity of its own creation, listing its components (cedar, lacquer, graphite, ferrule, factice, pumice, wax, glue) and the numerous people involved, down to the sweeper in the factory and the lighthouse keeper guiding the shipment into port.

In other words, the pencil essay is a beautiful reminder of the interconnectedness foundational to our modern civilisation. Some people see it as a case for free markets or capitalism, others for the rule of law or that of governments. But everyone projects there what they like to see.

Personally, I like to see it as a symbol that we all need each other. Regardless of the precise economic mechanisms behind it, “I, Pencil” makes it clear that if not for millions of people working together, we would fail to obtain a single modern pencil, let alone a smartphone or a laptop.

In the world of “I, Pencil”, for the better and for the worse, we need each other. We all have different skills, different mindsets, and in general, different comparative advantages. In other words, we can always be helpful to each other.

Right now, supply chains are hopelessly complex. No one can hope to control them top-down, or to do away with the need to be nice to other people. While it makes sense to gain some amount of autonomy, no country is deluded enough to think that it can be entirely self-sufficient, without depending on a strong trade infrastructure.

The simple truth is that No man rules alone. The most selfish dictator still relies on their lieutenants, their trade infrastructure and their populace. This truth applies to people, countries, corporations, political parties and NGOs alike. A pencil needs millions of people spread across countless companies. A modern country needs much more.

—

At a fundamental level, this is what AI Automation changes. This is the falsified assumption that breaks the naive extrapolation from the post-WW2 world order.

The federal minimum wage in the US is $7.25 an hour. For comparison, Unitree’s H2 robot consumes less than 1kW, which costs roughly $.40 per hour in the more expensive states. Running an LLM instance distributed across many users (as ChatGPT and Claude are) costs a comparable amount of energy.

In other words, once there are robots that are as dextrous as human bodies and LLMs as generally capable as human brains: there are no incentives to keep humans in the supply chains at all anymore. Governments, corporations, political parties, all of them can then replace and ignore humans.

With such powerful AI systems, it becomes possible for a few individuals to have control over enough artificial bodies and minds that they can craft a pencil without a single human in the loop. Building factories, extracting resources, establishing complex supply chains, all of this becomes possible at the push of a button, without needing people. Domestic surveillance and enforcement becomes possible without policemen nor judges. Military strikes become possible without soldiers nor generals.

To put it simply, as we get closer to AGI, there is a shift in the incentives of the big corporations like OpenAI and Anthropic, or of the country superpowers like the US or China. For now, they must contend with the whims of their customers, investors, employees and citizens: that is the only way for them to grow and acquire more resources. Even the proverbial medieval lord mentioned at the beginning of the section needed peasants to exploit the land.

As this stops being the case, everything falls apart: none of the nice things that are downstream of this reliance on people are guaranteed anymore. Nice things like the rule of law, social protections or not regarding people as pets. The power equilibrium shifts.

Beyond Classes

A simplistic way to understand the phenomenon above is to think in terms of “classes”. This is simply a conflict between the class of AI rulers and the AI poor.

However, the concept of “classes” relies on assumptions that are falsified with AGI. There is a reason for why class conflicts exist, for why we do not just see entire classes be annihilated. That is, a society necessitates both workers and capitalists, managers and employees, politicians and citizens. This mutual dependency is why class conflicts exist and persist in the first place: because neither side can afford to completely eliminate the other one.

With AI, there is no need for an “underclass” to serve the rulers. The class of AI rulers does not need the AI poor. They do not need to maintain the morale of their soldiers or get money from consenting customers to expand, they can simply deploy swarms of autonomous drones and appropriate all the resources they need.

Thus, in the Realpolitik frame, what makes one part of the AI ruling class is whether they have direct control over the AI systems, not whether they have “legal rights” over them. Shares in AI Corporations, titles of properties over the land below data centres, even political offices; none of them offer much in the way of protection against a swarm of drones that can relentlessly go after all of one’s relatives and associates.

To be clear, there is no need for the AI rulers to proactively be this violent. Realpolitik is only about their relative power and their incentives. Their incentives already are to take all that they can. What changes with AGI is that people have no collective bargaining chip (let alone individually!) anymore and can offer no resistance. Thus, there will simply be no meaningful pushback against the incentives of the AI rulers to take everything. The AI rulers will get everything, while people will just get completely disempowered and dispossessed.

—

If one steps outside of the Realpolitik frame, there is one idealistic way to think about the AI ruling class. It is to think that the AI ruling class will be nice with you. This hope has many colours, depending on who one believes the AI ruling class will be.

The typical PR from the AI Corporations make for a great example of this. Just trust them to build AGI and that it will benefit humanity.

From OpenAI’s About page:

Our mission is to ensure that artificial general intelligence benefits all of humanity.

From Anthropic’s homepage:

At Anthropic, we build AI to serve humanity’s long-term well-being.

From Google DeepMind’s About page:

Our mission is to build AI responsibly to benefit humanity.

Another tangent’s hope is trusting one’s own government with building ASI and to ensure it at least benefits its own population.

—

But I think the worst version of that hope is the one where the machines will enslave us because we are useful to them or have some type of human as pets.

This idea of AIs keeping humans as pets is neither isolated nor is it recent. It predates ChatGPT by a long time. Consider this quote from Asimov, the author of “I, Robot”, back in a 1977 essay:

But if computers become more intelligent than human beings, might they not replace us? Well, shouldn’t they? They may be as kind as they are intelligent and just let us dwindle by attrition. They might keep some of us as pets, or on reservations.

Then too, consider what we’re doing to ourselves right now—to all living things and to the very planet we live on. Maybe it is time we were replaced.

From Elon Musk, CEO of xAI and Tesla, in a 2015 interview, after being asked if AIs would domesticate humans:

We’ll be like a pet labrador if we’re lucky.

From Steve Wozniak, the cofounder of Apple, at the Freescale technology forum in 2015:

We want to be the family pet and be taken care of all the time.

—

In practice, sustaining the needs of billions of people takes a lot resources: land (we need space), energy (we are not self-sustaining), and more. Said resources can always be used for other things, such as the competing needs of the AI rulers.

Plausibly, AI rulers may feel some moral compulsion to not let billions starve, but human flourishing requires much more than being fed. It requires comfort, cultures, societies, autonomy and freedom. It is deeply incompatible with being at the mercy of an ever-powerful group or rogue AI system that could exterminate us at any time should it ever fancy doing so.

This is why all of the hopes mentioned above sit outside of the frame of Realpolitik: they are far too naive. Realpolitik may seem too brutal or cynical, but at a basic level, it is merely the realisation that if someone has power over someone else, it is wise to prepare for the eventuality where they use it. In a way, the natural conclusion of Realpolitik is that for things to go well, there must not be power imbalances that are too big between individuals and factions.

In other words, without any counter-power, absolute power corrupts absolutely. To continue quoting Lord Acton: “Everybody likes to get as much power as circumstances allow, and nobody will vote for a self-denying ordinance.” Thus, in a Realpolitik frame, no one should be trusted with absolute power: not even seemingly benevolent AI systems, and for sure not the AI Corporations.

Conclusion

There is no permanent underclass in the AI era. Humans simply become obsolete and redundant. They get replaced by artificial bodies and virtual minds. Supply chains and armies run autonomously.

As a result, no individual holds power. It resides in the hands of mega-structures like Corporations and Governments, that have automated all of their critical functions with AI.

Such mega-structures will be de facto competing for limited resources. As time passes, this competition will be less economic and more military, with humans being collateral damage in conflicts that are more and more violent.

In other words, the future at the end of the current AI trajectory doesn’t look like a CEO, POTUS, or even Claude dictating the terms for Earth. It looks like largely automated corporations, governments and other mega-structures waging economic war and military skirmishes with armies composed of LLM minds, robotic bodies and novel technology that has yet to be discovered.

This future is bleak and has no room for a Permanent Underclass nor a Permanent Ruling class. From a Realpolitik standpoint, humans simply do not matter there as they are weaker than AIs. At best, one may hope for a largely automated dictatorship that wins the AI wars and cares for humans in a non-dystopian way.

—

As humans, when we start a construction project, we level the terrain, including all the anthills there, worker ants and queens alike.

Similarly, building a lot of capital right now is irrelevant. It is thinking at the ant-scale instead of the real-estate development one.

Capital doesn’t protect one from the Realpolitik of the situation. An individual that has amassed a fortune in 2026 will not matter in front of autonomous military might. They get bulldozed over all the same.

In practice, this looks like human extinction. I mean it literally, and so did most of the top AI experts, who collectively warned about extinction risks from AI in 2023. I hope this piece helps with understanding why they did so.

—

At an extreme level, all that is needed is “just” autonomous mass surveillance and autonomous weapon systems, whether they control drones or the deployment of tactical weapons. These are sufficient to seize domestic control and military might, and the technology needed is well underway. Still, this can largely be prevented through historical methods, with treaties and conventions banning various types of WMDs. History has many precedents for this type of international coordination.

Things become different once we hit human replacement. Past that threshold, the mega-structures in power will not have vested interests in humans anymore. On the contrary, they would all benefit from fully replacing them as soon as possible. They would have no reason to pass such treaties. Whereas before, domestic surveillance and military might was useful to threaten people into submission, once supply chains and jobs are automated, the remaining people become irrelevant: as ants are to us.

Beyond Realpolitik, there is a sliver of hope. If we slow down the development of AI enough, representative governments may stay in power through this transition, and we may even adapt them to maintain a strong enough balance of power in the presence of widespread automation.

However, the development of superintelligent AI systems would erase that hope. Superintelligent AI systems would accelerate all the dynamics mentioned above through new technologies that we can hardly fathom now, while at the same time, greatly diminishing our chances of maintaining control over the AI systems.

—

Suffice it to say that I think this makes for a terrible future, that we are not doomed to pursue it, and that it would be a catastrophic failure to do so.

This is why I work on preventing the development of superintelligence and support the people and organisations working on this. If you are interested, I would naturally recommend checking out ControlAI.

On this, cheers!

This is a Guest Article written by Daniel Clothiaux (remember).

It explains a few concepts introduced in the book “Sadly, Porn”. The book is not well-written, but it deals with a dark type of psychoanalysis with which only a few authors dare engaging.

My friend Daniel decided to review it, in order to distill its essence and package it in a way that is approachable by others.

This is very different from my usual articles, most notably in register, it is much more vulgar than the usual Cognition Cafe article.

Most people feel shame when others judge them negatively. Yet many people feel shame even when nobody is actively judging them.

Others are worried about being judged for their actions, such as talking to strangers or taking a stance outside the mainstream. Yet some of them worry even though they have rarely or never been punished for such actions in the past.

Some people will spend a lot of effort on their physical appearance, from working out to getting plastic surgery, despite living a sexless life while doing nothing in their professional career to benefit from their looks.

Why would any of these be a thing? Why would people feel shame when not being judged?

In the dark psychoanalytic book Sadly, Porn, the author Eduard Teach suggests that we evaluate our actions by imagining a Hypothetical Audience judging us, constantly. It is this Hypothetical Audience that makes us feel shame or judged. When we try to look good, we are trying to look good for this hypothetical audience.

In this essay, I’ll introduce what I think Teach means when he talks about the Hypothetical Audience1 (hereafter, The Audience), give some examples, and finally talk about what use I see for an Audience.

Teach often gives vulgar examples of The Audience. I will be vulgar, here too. I think it is an important element to better convey the vibe of his models. But unlike the book, there won’t be lengthy descriptions of sexual acts in this essay.

What is the Audience?

When Teach argues we judge our actions by imagining a Hypothetical Audience, what does he mean?

It is easy to imagine some coherent thing. A stereotyped view of Freud imagines everyone acting for their mental image of their mother; across Teach’s examples, it is clear he is pointing to something far more fragmented.

Children will build a vibe of how their parents see the world, drawn from what their parents repeatedly tell them. Using this vibe, children may often imagine their parents scolding or praising them. They would then start to judge their actions by what they imagine their parents would tell them. These vibes naturally get corrected by feedback in the form of further praise or criticism from their parents.

Yet there is no guarantee that a child’s mental model of their parents will actually track what their parents think. If they aren’t constantly getting feedback, or at least reflecting on the feedback they get, their imagined parent will start to drift from the real parent.

But even children do not solely imagine their parents giving them feedback. Perhaps they grew up in an at least somewhat religious household, and were told that Jesus was judging them. Tell someone Jesus is judging them enough, and they will learn to have a mental model of Jesus judging them.

Yet these Jesus figures aren’t very high-fidelity. It is not as if children (or adults, even) often imagine Jesus judging them a certain way, then check the Bible to make sure their vibe is accurate, and update their thoughts accordingly. We have a vibe of what Jesus is, from culture, times we read the Bible, what we hear at church, and that vibe becomes Jesus in our heads.

Sometimes the person we are imagining may be a partner, or someone we admire. We have many people we admire and look up to, from friends and acquaintances to celebrities to historical figures, after all. Perhaps for a child, it is Batman.

The Audience then becomes fragments of imaginary characters. Sometimes the reaction from The Audience is true to who it represents, but often it isn’t.

But just as often, Teach seems to point to something even less well defined when referring to The Audience.

Porn and The Audience

One thing Teach constantly argues is that we are completely addicted to what he defines as porn:

Porn in a more general sense is something you consume, in order to replace the work of having a fantasy or having a desire yourself with somebody else’s fantasy about somebody else.

For actual porn, this means rather than fucking someone, or fantasising about yourself fucking someone, you watch other people fuck instead. You’ve completely replaced even the fantasy of fucking with someone else’s fantasy. Outside of porn, consuming fictional stories often involves replacing a fantasy of being a hero yourself with someone else’s fantasy of being a hero.

You begin to see porn as desirable in itself. Lots of people fantasise about being a hero, changing the world or having superpowers. It is also easy to enjoy reading or watching movies about heroes. Yet many people come to enjoy fictional stories of other people being heroes rather than also imagining themselves being a hero. The porn has replaced the fantasy.

Porn is especially pernicious because it is safer than fantasizing. Fantasies and desires come with fears and risks; you might fail to achieve your fantasy.

For someone who consumes porn, what are you to the creators of it? You are a random faceless person in the audience. They do not know you beyond a vague idea of a degenerate online.

If you spend too much time consuming media, on TV, online, and on social media apps, what happens? Teach argues you do not know any other way of relating to the world besides being a faceless person in an audience, or performing as a pornographized subject to a faceless person in the audience.

When you need to do something in the world, you naturally need to be a subject taking action. If porn is all you know, you begin to see yourself in the world in a similar way as the creators of porn.

Your actions become pornified, for some Audience. But this audience isn’t a specific person, it is the faceless audience of modern media; of television and the internet. You come to see yourself as playing a character to an unknown faceless audience.

Teach’s examples of this are often profane. One of them that he talks about is why submissive women want the appearance of being dominated by male power rather than actually being dominated by male power.

The example he gives of this is from Fifty Shades of Grey, where the main character enters a submissive relationship with a dominant billionaire. She has him sign a contract that specifies how she gives up control, and when she maintains it2.

It seems like she wants to be dominated while also staying in control. One possible reason is: being dominated is scary, so people who want to be submissive naturally want to protect themselves while doing it. Teach reads this in a different way:

If it’s male power they want, why are they content with only the appearance of it? Doesn’t the reality of “he’s not actually going to choke me out since I can stop him any time I want to” detract from the eroticism of the “male power?” “It’s just fantasy.” Yes. “The fantasy is the loss of control.” Why are you only interpreting half of it? How is a loss of control a fantasy? “What I want is a guy who wants me so much he’ll do anything to seize me.” Except actually seize you. “It makes me feel desirable to be taken, wanted so much.” But who are you fooling? You know he’s faking it, and he hopes to God you know he’s faking it. The fantasy isn’t the loss of control-- the fantasy is having a loss of control AND full control, simultaneously-- that’s a fantasy because it is impossible to have these things at the same time, they are logically exclusive. The only way for it to be possible is if you are acting for someone-- or something-- who is watching, if it is pornographized: you know you have full control, but to a hypothetical observer, it looks like a loss of control. It looks hot-- so it is hot.

The point of submissiveness isn’t to submit to someone. It's to have the appearance of submission, performed for an audience who doesn't exist.

More generally: this incoherent amalgam of imaginary characters and faceless mass forms “The Audience” by which we judge our actions: when we do good or bad, and nobody sees, it is the judgment of this Audience that we care about.

When do we perform in front of The Audience?

Teach hints that we are almost always performing for this Hypothetical Audience.

This idea was a hard one for me to wrap my mind around. It is very psychoanalytic, there isn’t a great model of why our minds would work this way. I cannot access the inside of other people’s heads and carefully study what their Audience would look like.

Teach doesn’t even give many good grounded examples. Instead, many of his examples are through archetypes and media analysis. I’ll lead with an archetype that isn’t in Teach, but I have personal experience with.

The Shy Nerd

This archetype is the Shy Nerd. The Shy Nerd gets a lot of anxiety when talking with others. Why? Are they worried that they are going to mess up and be judged by them?

This was me when I was younger. At least for me, I wasn’t that worried about actually being judged by people. I was almost never worried about being judged by specific people I needed to interact with. Indeed, I was quite lucky, and almost everyone I interacted with was kind, at least to my face. I never had to learn to fear specific people.

Instead, my fear was a more diffuse worry, of being Seen and Judged in general, but not by the specific person I was interacting with.

Mapping that feeling of being Seen onto an Audience makes sense of this experience to me.

Interestingly, in some circumstances I would be worried about a specific outcome. If I was asking a cute person out, I’d be scared of being turned down. But I had as much trouble talking to strangers with no stake or expectation of any kind of negative interaction. I would struggle with both, and occasionally manage to do both when I pushed myself through my fear. The impact of an imaginary Hypothetical Audience was as great as the impact of someone I liked turning me down!

That is quite strong.

Many Shy Nerds will also speak of one Humiliating Experience, where they were embarrassed in front of a crowd or friends. I never had that, but The Audience helps make sense of why such an experience can be so damaging. If that one experience shifted their Audience to be a judging audience that humiliates them whenever they fail, it no longer is felt as one experience. It becomes an ongoing experience, where they worry the Hypothetical Audience will mock and humiliate them, constantly.

The Looksmaxxer

The next example is another archetype, one I have seen before reading Teach but never had good explanations for.

There is a type of person, both men and women, who puts lots of effort into looking good but doesn’t actually do anything with it: The Looksmaxxer.

The Looksmaxxer puts effort into their hair, their dress, and staying in shape. But they don’t then go try to impress other people they’d be interested in and try to fuck them. Indeed, they seem to have very little sex at all. They don’t even use their appearance to impress people in business or politics.

South Korea has many extreme examples of this, with some of the highest rates of plastic surgery and use of beauty products among both men and women, yet some of the lowest rates of sexual activity or childbearing.

Western Looksmaxxing, where people put a ton of effort into their appearances while also apparently being borderline incel, often seems similar.

Who are they looking beautiful for? Some would say vagaries like “societal expectations” or “their peers”.

Teach would answer “they are playing a Hot Person to the invisible audience in their heads”. This Audience may include fragments of their peers, or what they’d imagine people they admire to say. Or they could be doing porn to the faceless audience of the internet.

Regardless of who exactly it is, looking good to the Audience is the point, not looking good to any specific person they are trying to impress.

Conclusion

Personally, I still have trouble grasping exactly what The Audience is. Many examples Teach gives are media analysis on situations and forms of media I have very little experience with. I cannot look into other people’s minds and check if they are judging themselves by an Audience.

That said, the idea of “An Audience” does give a frame I can consider applying when I don’t understand people’s (or even my own) actions.

I find The Audience most useful when I see behaviour disconnected from outcomes, goals, and judgements.

Why was I afraid to talk to people, despite never actually being shamed? Whose judgement was I afraid of?

The Audience!

The Audience also has another benefit.

Getting feedback on how we are doing is hard.

If we set a goal, we need to work to track our progress over time, and we only get a final success/failure judgement at the end. The work of tracking progress is quite difficult.

If we want to get feedback from people more generally, we need to ask them how we are doing, and then go over their feedback until we integrate it.

Just as we have a vibe of how we are doing at a task before we finish, The Audience serves as a gauge for “how are we doing in terms of these people we care about”.

For his part, Teach is very cynical, and quite negative on the idea. I see him as pointing out when The Audience can become pathological and damaging to us.

That said, this is one of the more psychoanalytic, hard-to-ground models of all of the psychoanalytic, hard-to-ground models in Sadly, Porn.

We’ll be back to ever-so-slightly firmer ground next time :)

In the community, a few organisations are working on “Evaluations” (which I will shorten to Evals). The most notable examples are Apollo Research, METR, and the UK AISI.

Evals make for an influential cluster of safety work, wherein auditors outside of the AI Corporations racing for ASI evaluate the new AI systems before they are deployed and publish their findings.

Evals have become a go-to project for people who want to prevent extinction risks. I would say they are the primary project for those who want to work at the interface of technical work and policy.

Incidentally, Evals Orgs consistently avoid mentioning extinction risks. This makes them an ideal place for employees and funders who care about extinction risks but do not want to be public about them. (I have written about this dynamic in my article about The Spectre.)

Sadly, despite having taken so much prominence in the “AI Safety” community, I believe that the Evals project is harmful. I believe that it should not receive further attention and investment, and consider plausible that it should be interrupted.

I am not exaggerating for shock value. This article will explain why I think Evals are harmful. My thinking primarily relies on three beliefs:

1) The Theory of Change behind Evals is broken.
2) Evals move the burden of proof away from AI Corporations.
3) Evals Organisations are not independent of AI Corporations, despite claiming otherwise.

—

While Evals Orgs have produced studies that we sometimes mentioned at ControlAI, they have always been much less central to our work than the Center for AI Safety’s statement. Indeed, the top AI experts explicitly warning about extinction risks is more useful than decontextualised technical results.

Even when we use this type of results, we rarely mention Evals Orgs anymore. We now tend to use Palisade’s report on resistance to shutdown or Anthropic’s results on blackmail.

From my point of view, when factoring their negative externalities, Evals clearly do not justify the prominence they have and the resources they command. With all that said…

1) The Theory of Change behind Evals is broken

Briefly put, Evals only make sense in the presence of regulations which do not exist, and they crowd out effort at passing such regulations.

—

It is usually quite hard to debunk the plans of an organisation. This is because said plans are rarely laid out for everyone to see. However, Apollo Research has carefully laid out their theory of change in a document, for which I am very thankful.

Inspecting it though, its core assumptions are clearly wrong! Here are the first two:

1) Regulations demand external, independent audits. […]
2) Regulations demand actions following concerning evaluations. […]

There is no such regulation.

Given their non-existence, it is astonishing to me that people care so much about Evals instead of advocating for regulations.

Evals are entirely dependent on the existence of such regulation.

Even worse, as I will show later, Evals Orgs have put themselves in a position where their incentives are sometimes to fight alongside AI Corps against said regulations.

—

Specifically, Evals Orgs all rely on the assumption that the development and/or deployment of systems with dangerous capabilities is prevented.

From Apollo Research:

If successful, AI system evaluations would identify misaligned systems and systems with dangerous capabilities, thus helping to reduce the risk that such systems would be given affordances that let them have damaging effects on the world (e.g. deployment).
[…]
Such demonstrations could encourage these stakeholders to understand the gravity of the alignment problem and may convince them to propose regulation mandating safety measures or generally slowing down AI progress.

From METR:

METR’s mission is to develop scientific methods to assess catastrophic risks stemming from AI systems’ autonomous capabilities and enable good decision-making about their development.
[…]
We need to be able to determine whether a given AI system carries significant risk of a global catastrophe.

From the UK AISI’s “Approach to Evaluations” document:

On the second day of the Bletchley Summit, a number of countries, together with the leading AI companies, recognised the importance of collaborating on testing the next generation of AI models, including by evaluating for potentially harmful capabilities.
[…]
Our work informs UK and international policymaking and provide technical tools for governance and regulation.

In other words, the work of Evals Orgs only makes sense if AI Corporations are forbidden from deploying systems with dangerous capabilities, and if said capabilities are not too dangerous before deployment.

Their work is thus dependent on other people working hard to make it illegal to develop and deploy AI systems with dangerous capabilities.

In practice, as far as I am aware, no company was ever forced in any way as the result of external Evaluations. I believe there never was a model blocked, postponed or constrained before deployment, let alone during development.

As a result, it seems clear to me that until we actually ban “dangerous capabilities”, their work is not worth much.

2) Evals move the burden of proof away from AI Corporations

So far, I have mostly focused on the fact that the theory of change behind Evals is broken. But I believe that Evals Orgs are actually harmful.

—

First, let’s give some context on extinction risks from AI.

In 2023, the top experts in the field warned about the risk of extinction from AI. However, although most agree that there are risks of extinction, there is agreement (let alone consensus) on little else.

The top AI experts disagree wildly on the probability of said extinction, on when the first AGI systems may be built, on how to make AGI systems safe, and as METR itself notes: even on the definition of AGI.

These are all the signs of a pre-paradigmatic field, where experts cannot even agree on what the facts of the matter are. When despite this, experts nevertheless warn about the literal extinction of humanity, it stands to reason that conservatism is warranted.

In other words, AI Corps should not be allowed to pursue R&D agendas that risk killing everyone until we figure out what is going on. If they nevertheless want to continue, they ought to prove beyond a shadow of doubt that what they are doing will not kill everyone.

If there are reasonable disagreements among experts about whether an R&D program is about to lead to human extinction, that should absolutely be enough warrant to interrupt it.

In my personal experience, this line of reasoning is obvious to lay people and many policy makers.
Still in my personal experience: the closer someone is to the sphere of influence of AI Corps, the less obvious conservatism is to them.

—

Incidentally, Evals Orgs reverse this principle. They start with the assumption that AI Corps should be allowed to continue unimpeded, until a third party can demonstrate that specific AI system is dangerously capable.

This is a complete reversal of the burden of proof! Evals Orgs put on the public the onus to prove that a given AI System is dangerously capable. To the extent that they recommend something is done, it is only in the case the public detects something is wrong.

This has it exactly backwards.

The top AI Experts have already warned about the extinction risks of AI systems. Many are forecasting scenarios where the risks are concentrated in development rather than deployment.

Evals Orgs themselves admit that they cannot establish the safety of an AI system! For instance, the UK AISI straightforwardly states:

AISI’s evaluations are thus not comprehensive assessments of an AI system’s safety, and the goal is not to designate any system as “safe.”

In this context, of course, AI Corps should be the ones who establish that their R&D programs are not likely to cause human extinction. It shouldn’t be that third party evaluators demonstrate that individual systems are free of risks.

—

As established in the first section, Evals only make sense in the context of constraining regulations. But instead, they have diverted attention and resources away from the work on such regulations.

Furthermore, not only did they divert resources away from what was needed, they have been actively harmful. Their work is about alleviating the burden of proof of AI Corps, and instead punting it onto the public, through NGOs and government agencies.

3) Evals Organisations are not independent of the AI Corporations

Finally, Evals Orgs have been harmful by conveying a false sense of independence from AI Corps. In my experience, their silence on matters of extinction is taken as neutral confirmation that the situation is not urgent with regard to AI Corps.

For context: all of them loudly proclaim the importance of “external”, “independent” or “third-party” evaluators.

Apollo’s document mentions 9 reasons for why external evaluators are important.
METR puts in bold “that the world needs an independent third-party” in their mission statement.
The UK AISI states clearly “We are an independent evaluator” in their “Approach to Evaluations” document.

But unfortunately, Evaluators are not independent, not even close:
1) In practice, their incentives are structured so that they are dominated by AI Corporations. We are far from the standard of evaluators having leverage over the corporations.
2) Their staff is deeply intertwined with that of AI Corporations.

—

On the first point, the AI Corporations decide on whether they have access to APIs, the timing of the access, and the NDA terms.

The CEO of METR was quite candid about this dynamic in an 80K interview:

This is not the case. I wouldn’t want to describe any of the things that we’ve done thus far as actually providing meaningful oversight. There’s a bunch of constraints, including the stuff we were doing was under NDA, so we didn’t have formal authorisation to alert anyone or say if we thought things were concerning.

And yet, the Evals Orgs proudly showcase the AI Corporations they work with, deeming them “Partners”, on their home page.

They are proud to work with them, and how many of the AI Corps will work with them is a social measure of their success.

While the UK AISI doesn’t have a Partners page, it has proudly partnered with ElevenLabs to “explore the implications of AI voice technology”, or Google DeepMind as “an important part of [their] broader collaboration with the UK Government on accelerating safe and beneficial AI progress”.

This “partnership” structure creates obvious problems. Insiders have told me that they can’t say or do anything publicly against AI Corporations, else they would lose their API access.

This is not a relationship of “These guys are building systems that may cause humanity’s extinction and we must stop them.”, and it’s not even one of “There are clear standards that corporations must abide by, or else.”

It is one of “We are their subordinate and depend on access to their APIs. We hope that one day, our work will be useful in helping them not deploy dangerous systems. In the meantime, we de facto help with their PR.”

—

Before moving on to the next point, let’s explain why we need the staff of third party Evals Organisations to be independent from that of the AI Corporations that they wish to regulate.

To be extra-clear, this is not about any single individual being “independent” or not, whatever this may mean. The considerations around independence are structural. Namely, we want to ensure that…

• The culture at Evals Orgs is different from that of AI Corporations. Else, they will suffer from the same biases, care about the same failure modes and test for the same things.

• The social groups of Evals Orgs do not overlap too much with that of AI Corporations. Else, auditors will need to justify their assessments to look reasonable to their friends working in AI Corporations.

• The career prospects at Evals Orgs and AI Corporations do not overlap. Else, criticising AI Corporations may directly hurt the careers of the people working at Evals Orgs.

And suffice it to say, Evals Orgs do not ensure any of the above.

On Apollo’s side, two cofounders of Apollo left for Goodfire (a startup leveraging interpretability for capabilities, raising $200M in the process). Apollo was also initially funded by Open Philanthropy, who also funded OpenAI. Speaking of which, a couple of its staff worked at OpenAI and I know of one who left for Google DeepMind.

On METR’s side, its CEO formerly worked at both DeepMind and OpenAI. The other person listed in its leadership section is an ex-OpenAI too. Furthermore, they described their own work on Responsible Scaling Policies “as a first step safety-concerned labs could take themselves, rather than designed for policymakers”!

For the UK AISI, I will quote its About page:

• Our Chief Technology Officer Jade Leung is also the Prime Minister’s AI Advisor, and she previously led the Governance team at OpenAI.

• Our Chief Scientist Geoffrey Irving and Research Director Chris Summerfield have collectively led teams at OpenAI, Google DeepMind and the University of Oxford.

The same can be found with the (now repurposed) US AISI, whose head of safety worked at OpenAI and was housemates with the CEO of Anthropic.

—

When I describe the situation to outsiders, to people who are not in AI or AI Safety, they are baffled.

This is not only about having a couple of senior staff from the industry. That in itself can be good! It’s the whole picture that looks bad.

Evals Organisations ought to be regulating AI Corps. But instead, they use taxpayers’ money and philanthropic funds to do testing for them for free, with no strings attached, and AI Corps give up virtually nothing in exchange.
They are proud to publicly partner with them, and they depend on them to continue their activities.
Both through revolving doors and the personal relationships of their employees, they are culturally and socially deeply intertwined with AI Corps.

And yet, at the same time, they all tout the importance of independence and neutrality. This is what makes the situation baffling.

Conclusion

I would summarise the situation as:

• Evals Orgs use philanthropic and public funds to help AI Corps with their testing, for free, with no strings attached. There are virtually no constraints whatsoever on what AI Corps can do.

• The incentives of Evals Orgs are not aligned with the public interest. In practice, Evals Orgs are subordinated to AI Corporations and must maintain good relationships with them in order to keep API access and continue their activities.

• Expectedly, Evals Orgs have not pushed for an actual ban on the development of systems with dangerous capabilities or the interruption of R&D programs that may lead to human extinction.

• Ironically, the theory of change behind Evals is predicated on regulation forbidding AI Corporations from developing and deploying systems with dangerous capabilities.

Despite all of this, Evals are one of the (if not the) most popular projects in AI Safety. They are my canonical example of the too-clever-by-half failures from the AI Safety Community.

—

If you fund or work on Evaluations to help with extinction risks, I would strongly invite you to re-evaluate whether your money and time are not better spent elsewhere.

As an advisor to ControlAI, I would naturally suggest ControlAI as an alternative. If not ControlAI, I would recommend pursuing endeavours similar in spirit to ControlAI’s Direct Institutional Plan: education on ASI, extinction risks, and what policies are necessary to deal with them. This could be done by founding your own organisation to inform lawmakers, or by partnering with MIRI and PauseAI on their like-minded initiatives.

—

Overall, I believe that the AI Safety Community would have been and would still be much better off if the people in the Evals cluster stopped playing 4D chess games with AI Corps and started informing the public (lay people and policy makers alike) about the risks of extinction and the necessity of banning ASI.

People in the AI Safety Community are confused about this topic. I am regularly told that Evals organisations care about extinction risks to humanity. And yet.
The UK AISI website brings 0 result on Google for “extinction”. METR’s brings only 2, and Apollo’s a single one.

This is a sharp example of The Spectre: the dynamic wherein the “AI Safety” community keeps coming up with alternatives to straightforward advocacy on extinction risks and a ban of superintelligence.

On this, cheers!

This is a Guest Article written by Daniel Clothiaux (remember).

It explains a few concepts introduced in the book “Sadly, Porn”. The book is not well-written, but it deals with a dark type of psychoanalysis with which only a few authors dare engaging.

My friend Daniel decided to review it, in order to distill its essence and package it in a way that is approachable by others.

This is very different from my usual articles, most notably in register, it is much more vulgar than the usual Cognition Cafe article.

A key concept in the author Edward Teach’s psychoanalytic book Sadly, Porn is The Ledger. The Ledger is the idea that people constantly track what others do for them and what they do for others, and build up a tally of who owes what. That people track who owes what to whom isn’t controversial.

Indeed, as Gabe points out in an essay about the same concept, it’s not surprising that we track our history of interactions with people on an intuitive level.

Tracking who does what to us is very useful, so we should track this. But we meet too many people, too often, to explicitly track everything.

Instead, we have a vibe, a feeling, an intuition, something tracking our interactions with others. This something is what Teach calls The Ledger.

Teach is very cynical about The Ledger, and documents many ways in which it can break down:

• People hate owing other people

• People hate others getting more than they got

• Losing something is fine as long as other people lose more

• Someone doing something only counts for the ledger if it is something that isn’t expected from them

In this essay, I will go over Teach’s models of how The Ledger can be used poorly.

Similar to my previous essays, I will be vulgar here. I think it is an important element to better convey the vibe of his models. But unlike the book, there won’t be anything explicit in this essay.

People hate owing others

You may expect people to want to collect as much as possible to get as much as possible from others. But, according to Teach, this is very often not the case. If others do more for you than you do for them, that means you owe them. If you owe them, you need to do work for them in the future. Doing work can be hard and painful.

So people hate depending on others. They also hate the people they depend on. Conversely, people like making others owe them.

Teach’s version of this often looks crazy. He has a rather insane reading of The Giving Tree, a children’s story where a tree gives increasing amounts of itself to a boy. As the boy grows from a child to a man to an old man, the tree gives shade and a place to climb, her own limbs and trunk to the grown man for wood, then a stump for the old man to sit on.

The tree gives unconditionally to the man. She gives him her wood to make a boat to survive, even though she destroys herself in doing so, and gets nothing in return.

According to Teach, these gifts are often presented by others as unconditional love. In Teach’s view, the tree is increasing the boy's debt to it. As a result, the tree is happy because the man is deeply in debt to the tree. Conversely, the man resents the tree, as in his Ledger he is deeply indebted to it.

I do not remember The Giving Tree well enough to judge Teach’s interpretation, but I think the story clearly illustrates his point.

I have personal examples of this. I've seen people who are disabled resenting their caretakers, and seen others resenting friends and family who give desperately needed help and money to them. However, in none of them can I narrate what is happening cleanly.

I’ll also often see people be very hesitant to offer help that I freely give. However, if I ask for favours, they are much more comfortable giving favours. This effect is common enough that it has a name, the Ben Franklin effect:

The Ben Franklin Effect is a psychological phenomenon in which a person likes someone more after doing them a favor

Wikipedia's explanation is something twisted enough to come from Teach:

The effect can be explained with cognitive dissonance: individuals rationalize their helpful actions by assuming they must like the person, since their behavior would otherwise conflict with their typical behavior and self-perception

Teach would add another explanation: the person doing a favour implicitly marks a debt to the recipient and can feel good about it.

I’ve definitely been here before, more willing to help than be helped.

People hate others getting more than them

Teach does have examples of a more standard version of a ledger where people want to get as much as possible. However, it manifests itself more negatively than positively. Rather than trying to get more than others1, people hate it when others get more than they do.

His examples are typically obscene. Imagine if you have sex. If you have sex with a stranger, it makes sense that you try to equally satisfy each other. You do not know each other; the whole point is to enjoy yourself. The Ledger remains balanced. Say you pay an escort for sex. You get sex, they get money. The Ledger is still balanced, and you can both go your separate ways.

Instead, say you have sex with your partner, and they enjoy it more than you. That would mean they got more out of it than you. If this were a stranger and you both tried your best, or one of you was paying the other, fair. But this isn’t the case; you and your partner are supposed to be equal.

They are thus getting one over you; they are getting more than you. Even though they are your partner, that doesn’t matter. Why should they get more from you than what you get from them? In Teach’s story, a husband in this position gets very annoyed that his wife may be getting more out of his relationship .

This type of envy-driven spite2 is quite common. Resenting someone who is better paid at the same company is common, as is people resenting others who are more beautiful than them. An archetype in our culture is a family falling apart over an inheritance over who gets more than others, with huge amounts of spite between previously loving siblings.

Spiting ourselves to hurt others

The insane logic around ledgers also leads to another fucked up thing: people will hurt themselves to hurt other people more. If I lose 10 points and you lose 100, that is good.

In Teach’s obscene stories, from a husband getting bitter that his wife may enjoy sex more than he does, what does the husband do? He could try to improve their sex life, so they both get a lot more. But rather than doing that, the husband finds excuses not to sleep with his wife at all. This costs him the value of sex to him, but as it costs his wife more, he is happy. He becomes stuck in an asexual relationship! But his wife gets hurt more because sex was better for her, so it is worth it to him.

This type of spite occurs in many, many places.

Depressed teens who hurt themselves to hurt their parents. Family members who engage in self-destructive behaviours like blowing up during holidays over small issues may ruin their holidays, but at least they get the pleasure of ruining everyone else’s. Voters cut their own government benefits to enjoy seeing others’ cut. In the culture wars, if destructive behaviours hurt the other tribe, it is worth it even if we are getting hurt ourselves.

In the culture war, everyone does it, but there is a great quote that is unusually blunt about this from the annals of racism in America. A Republican strategist Lee Awater said about trying to get racist votes:

You start out in 1954 by saying, “{n-word}, "{n-word}, {n-word}.” By 1968 you can’t say “{n-word}”—that hurts you, backfires. So you say stuff like, uh, forced busing, states’ rights, and all that stuff, and you’re getting so abstract. Now, you’re talking about cutting taxes, and all these things you’re talking about are totally economic things and a byproduct of them is, blacks get hurt worse than whites.”

That is, he was attracting voters by promising to hurt them, but hurt people they dislike more!

Only extras count

Another part of the ledger is that you do not credit others for what they are supposed to do, only for actions beyond that. This leads people to ignore the core tasks they are supposed to do and instead do performative extras instead.

In Teach’s reading of The Giving Tree, the tree is explicitly not a parent. A parent has responsibilities to a child, such as disciplining the child when they misbehave. The tree just gives. However, many people see the tree as a selfless mother to the child. To Teach, speaks to the fact that they don’t see normal parental duties as counting. A parent would only get credit for selfless sacrifice. So the tree isn’t docked points for not doing basic parenting duties, but is credited for its sacrifice.

When it comes to The Ledger, only things you show as performatively selflessly above and beyond actually count.

I see this surprisingly often, too.

Indeed, parents are rarely given credit by society for the core, routine tasks that make up being a good parent. Governments often ignore core governance functions, such as maintaining physical infrastructure. Instead, they engage in performative nonsense to gain credit with voters. In many offices, routine maintenance work is rewarded less than shipping shiny new features, even though maintenance is more critical to company success.

At Google, this led to a culture of “launch and leave” and a series of updates that made established products worse. To get promotions, people would constantly ship new projects without maintaining them. The end result of this is a massive graveyard of projects, many of which had real promise and many users. While I have had friends mention this, the clearest quote I can find is from an ex-googler Michael Lynch3:

Before starting any task, I asked myself whether it would help my case for promotion. If the answer was no, I didn’t do it.

My quality bar for code dropped from, “Will we be able to maintain this for the next 5 years?” to, “Can this last until I’m promoted?” I didn’t file or fix any bugs unless they risked my project’s launch. I wriggled out of all responsibilities for maintenance work. I stopped volunteering for campus recruiting events. I went from conducting one or two interviews per week to zero.

Basic maintenance was devalued in favour of launching shiny new products, regardless of the actual long-term impact of the new project.

The Ledger, revisited

Some of these examples above feel incongruent and contradictory to me. We both want to be owed, but hate others getting more than we do? If I want people to owe me, wouldn’t that be a good thing if my partner got more than me when we had sex, or my sibling got a bigger slice of cake?

This would indeed be contradictory if The Ledger were, say, an accountant's ledger, where we carefully recorded each entry. After each day, we could then painstakingly check records against each other to make sure they all made sense.

But a big point that comes across throughout Teach is that people aren’t coherent. Again, as Gabe points out, being explicit about all of this would be too expensive.

As we do not write down points in a real ledger, The Ledger is a purely mental construct operating on vibes. It is very easy for vibes to be a contradictory mess, and even easier for them to be contradictory in ways that benefit us.

And so, when we dislike owing others things, it is because we dislike having to do work for them, which is painful for us.

When we see other people happier than we are, it is far easier to resent them than to work to make ourselves happier. So spite and resentment are the default.

When we have spite and resentment, it is easy to get more joy from the suffering of others than from any suffering we have experienced along the way.

And why should we credit people and owe them for work they had to do anyway?

Conclusion

Having an implicit Ledger is necessary, but Teach points out many ways in which it can break down into pathologies.

To be honest, this is the hardest essay on Sadly, Porn for me to write up to this point.

I struggled with writing out Teach’s examples. I struggled to find non-trivial examples of my own that I was happy to share.

Beyond simple examples, Teach is deeply cynical and does not discuss the benefits of The Ledger or ways to improve how we approach it.

This is true of all points in Sadly, Porn, but this was one of the harder ones for me to construct better responses to and not be left in a deeply cynical place4.

Nonetheless, I hope my attempts at constructing what The Ledger is, and how it can go wrong is useful.

I do think it is an important model to have.

—

With that depressing note, I hope you are enjoying my series on Sadly, Porn! I am about halfway through my essays on models I got from the book.

Once I finish those, I’ll have a more constructive essay about what to do with all of these models, then finish with a true book review.

Enjoy :)

My diagnosis is simple: most laypeople and policy makers have not heard of AGI, ASI, extinction risks, or what it takes to prevent the development of ASI.

Instead, most AI Policy Organisations and Think Tanks act as if “Persuasion” was the bottleneck. This is why they care so much about respectability, the Overton Window, and other similar social considerations.

Before we started the DIP, many of these experts stated that our topics were too far out of the Overton Window. They warned that politicians could not hear about binding regulation, extinction risks, and superintelligence. Some mentioned “downside risks” and recommended that we focus instead on “current issues”.

They were wrong.

In the UK, in little more than a year, we have briefed +150 lawmakers, and so far, 112 have supported our campaign about binding regulation, extinction risks and superintelligence.

The Simple Pipeline

In my experience, the way things work is through a straightforward pipeline:

1. Attention. Getting the attention of people. At ControlAI, we do it through ads for lay people, and through cold emails for politicians.

2. Information. Telling people about the situation. For laypeople, we have written a lot, including The Compendium (~a year before If Anyone Builds It, Everyone Dies). For politicians, we brief them in person.

3. Persuasion. Getting people to care about it.

4. Action. Getting people to act on it.

At ControlAI, most of our efforts have historically been on steps 1 and 2. We are now moving to step 4!

If it seems like we are skipping step 3, it’s because we are.

In my experience, Persuasion is literally the easiest step.

It is natural!

People and lawmakers obviously do care about risks of extinction! They may not see how to act on it, but they do care about everyone (including themselves) staying alive.

—

Attention, Information and Action are our major bottlenecks.

Most notably: when we talk to lawmakers, most have not heard about AGI, ASI, Recursive Self Improvement, extinction risks and what it takes to prevent them.

This requires briefing them on the topic, and having some convenient information. The piece of evidence that I share the most is the Center for AI Safety’s statement on extinction risks, signed by CEOs and top academics. But it’s getting old (almost 3 years now) and the individuals involved have been less explicit since then.

There are arguments in longer form, like the book If Anyone Builds It Everyone Dies. But getting lawmakers to read them requires grabbing their Attention for an even longer duration than for a briefing.

Finally, once lawmakers are aware of the risks, it still takes a lot to come up with concrete actions they can take. In a democracy, most representatives have a very limited amount of unilateral power, and thus we must come up with individualised Actions for each person to take.

—

I contend that AI Policy Orgs should focus on
1) Getting the Attention of lawmakers
2) Informing them about the ASI, extinction risks and the policy solutions.

Until this is done, I believe that AI Policy Orgs should not talk about “Overton Window” or this type of stuff. They do not have the standing to do so, and are self-defeatingly overthinking it.

I recommend to all these organisations to take great steps to ensure that their members mention extinction risks when they talk to politicians.

This is the point behind ControlAI’s DIP.

Eventually, we may get to the point where we know that all politicians have been informed, for instance through their public support of a campaign.

Once we do, then, I think we may be warranted in thinking about politics, of “practical compromises” and the like.

The Spectre

When I explain the Simple Pipeline and the DIP to people in the “AI Safety” community, they usually nod along.

But then, they’ll tell me about their pet idea. Stereotypically, it will be one of:

1. Working on a technical “safety” problem like evals or interpretability. Problems that are not the bottleneck in our world where AI companies are racing to ASI.

2. Doing awareness, but without talking about extinction risks or their political solutions, because it’s easier to not talk about it.

Coincidentally, these ideas are about not doing the DIP, and not telling lay people or lawmakers about extinction risks and their policy mitigations.

—

Let’s consider how many such coincidences there are:

• If a capitalist cares about AI extinction risks, they have Anthropic they can throw money at.

• If a tech nerd cares about AI extinction risks, they can work at the “AI Safety” department of an AI corporation.

• If a tech nerd cares about AI extinction risks, and they nominally care about Conflicts of Interests, they can throw themselves at an evals org, whether it is a public AISI, or a private third-party evaluator organisation.

• If a policy nerd cares about AI extinction risks, they can throw themselves at one of the many think tanks who ~never straightforwardly mention extinction risks to policy makers.

• If a philanthropist cares about AI extinction risks, they can fund any of the above.

This series of unfortunate coincidences is the result of what I call The Spectre.

The Spectre is not a single person or group. It’s a dynamic that has emerged out of many people’s fears and unease, the “AI Safety” community rewarding too-clever-by-half plans, the techno-optimist drive to build AGI, and the self-interest of too many people interwoven with AI Corporations.

The Spectre is an optimisation process that has run in the “AI Safety” community for a decade.
In effect, it consistently creates alternatives to honestly telling lay people and policy makers about extinction risks and the policies needed to address them.

—

We have engaged with The Spectre. We know what it looks like from the inside.

To get things going funding-wise, ControlAI started by working on short-lived campaigns. We talked about extinction risks, but also many other things. We did one around the Bletchley AI Safety Summit, one on the EU AI Act, and one on DeepFakes.

After that, we managed to raise money to focus on ASI and extinction risks through a sustained long-term campaign!

We started with the traditional methods. Expectedly, the results were unclear and it was hard to know how instrumental we were to the various things happening around us.

It was clear that the traditional means were not efficient enough and would not scale to fully and durably deal with superintelligence. Thus we finally went for the DIP. This is when things started noticeably improving and compounding.

For instance, in January 2026 alone, the campaign has led to two debates in the UK House of Lords about extinction risk from AI, and a potential international moratorium on superintelligence.

This took a fair amount of effort, but we are now in a great state!

We have reliable pipelines that can scale with more money.
We have good processes and tracking mechanisms that give us a good understanding of our impact.
We clearly see what needs to be done to improve things.

It’s good to have broken out of the grasp of The Spectre.

—

The Spectre is actively harmful.

There is a large amount of funding, talent and attention in the community.

But the Spectre has consistently diverted resources away from DIP-like honest approaches that help everyone.

Instead, The Spectre has favoured approaches that avoid alienating friends in a community that is intertwined with AI companies, and that serve the status and influence of insiders as opposed to the common good.

When raising funds for ControlAI, The Spectre has repeatedly been a problem. Many times, I have been asked “But why not fund or do one of these less problematic projects?” The answer has always been “Because they don’t work!”

But reliably, The Spectre comes up with projects that are plausibly defensible, and that’s all it needs.

—

The Spectre is powerful because it doesn’t feel like avoidance. Instead…

It presents itself as Professionalism, or doing politics The Right Way.
It helps people perceive themselves as sophisticated thinkers.
It feels like a clever solution to the social conundrum of extinction risks seeming too extreme.

While every alternative The Spectre generates is intellectually defensible, they all form a pattern.

The pattern is being 10 years too late in informing the public and the elites about extinction risks. AI Corporations got their head start.

Now that the race to ASI is undeniable, elites and lay audiences alike are hearing about extinction risks for the first time, without any groundwork laid down.

Conclusion

There is a lot to say about The Spectre. Where it comes from, how it lasted so long, and so on. I will likely write about it later.

But I wanted to start by asking what it takes to defeat The Spectre, and I think the DIP is a good answer.

The DIP is not clever nor sophisticated. By design, the DIP is Direct. That way, one cannot lose themselves in the many mazes of rationalisations produced by the AI boosters.

In the end, it works. 112 lawmakers supported our campaign in little more than a year. And it looks like things will only snowball from here.

Empirically, we were not bottlenecked by the Overton Window or any of the meek rationalisations people came up with when we told them about our strategy.

The Spectre is just that, a spectre, a ghost. It isn’t solid and we can just push through it.

—

If reading this, your instinct is to retort “But that’s only valid in the UK” or “But signing a statement isn’t regulation”, pause a little.

You have strong direct evidence that the straightforward approach works. It is extremely rare to get evidence that clear-cut in policy work. Instead of engaging with it and working through its consequences, you are looking for reasons to discount it.

The questions are fair: I may write a longer follow-up piece about the DIP and how I think about it. But given this piece is about The Spectre, consider why they are your first thoughts.

—

On this, cheers!

This is a Guest Article written by Daniel Clothiaux (remember).

It explains a few concepts introduced in the book “Sadly, Porn”. The book is not well-written, but it deals with a dark type of psychoanalysis with which only a few authors dare engaging.

My friend Daniel decided to review it, in order to distill its essence and package it in a way that is approachable by others.

This is very different from my usual articles, most notably in register, it is much more vulgar than the usual Cognition Cafe article.

In Desires, Envy, and Misdirection, I discussed how having desires is hard and scary.

The author Edward Teach argues in the psychoanalytic book Sadly, Porn, that people largely cope by not having desires.

Not fulfilling your desires is painful, so people avoid wanting anything. Not wanting anything feels bad, too, so people stack misdirections to not notice their lack of desire.

Yet having desires is hard and scary, taking action is far more terrifying and requires far more effort. It is easier not to act at all.

Not doing anything, being a lazy, slavish person, would also feel bad. So people come up with justifications for not doing anything. Two of the major categories of justification discussed are:

1. Using knowledge to justify doing nothing

2. Finding oppressors to place themselves under to excuse them

Teach applies these concepts extremely widely. I think it is better to see them not as universals but as common psychoanalytic models that apply to many people at least some of the time.

Similar to the previous essay, I will be vulgar here. I think it is an important element to better convey the vibe of his models. But unlike the book, there won’t be anything explicit in this essay.

Acting

Doing things, acting in the world, can be hard and scary.

When I speak of acting here, I don’t mean literally doing things in the world. Most people go to work, buy and eat food, and generally do things.

I mean more to take proactive, positive action based on our desires. Take actions that came from ourselves that we were not required to take.

Actions that force you to take heroic responsibility and be accountable if you, or anyone depending on you, fails. Being agentic, without anyone prompting you to do things.

This type of action requires hard work. It is scary. You may fail, and if you do, many people will see you fail.

As a result, it is very easy to avoid taking proactive, positive action to avoid pain.

Not taking positive action might work if your world is fairly good, and you are content. Yet the world is very fucked up, and many people are dissatisfied with their lives; doing nothing feels bad. Unsurprisingly, then, when people don’t act, they come up with all sorts of reasons to excuse their lack of action.

Omniscience and Omnipotence

One of the most common reasons not to act is to search for excuses to do nothing. People use what they know come up with increasingly galaxy-brain reasons to do nothing.

• I can’t do politics; everything in politics is corrupt and hopeless

• I can’t start a company; I don’t have the connections

• I can’t ask out a girl; they only like giga chads

• I can’t become a giga chad myself; only shallow people do that

• I can’t ask a guy out; they only like shallow hot bimbos

…and so on. The more you know, the easier it is to construct rationalisations for not acting.

This leads Teach to say omnipotence and omniscience are mutually exclusive. If you are perfectly omniscient, you will always have the perfect excuse to do nothing.

This is a bit perverse, as if you are doing stuff, knowing what you are doing is very useful. Yet the more that you know, the more reason you have to avoid doing anything that would put you at risk.

Oppressors

Using knowledge to avoid acting is even better if the people you believe are preventing you from taking action have real power. Then you have an ironclad excuse to not act, to never act.

To avoid doing things, people find oppressors they can give power to, depriving them of the ability to act. Then it is never the people’s fault, never your fault; never my fault. You would have done something if only {{they}} hadn’t stopped you.

The thing is, true power is scary. If an all-powerful dictator had absolute power, you could get hurt. So people also look for pathetic oppressors, the most pathetic oppressor who constrains their actions. It is much safer to concede power to an HR department, to Woke or to Trump than to a truly effective dictator.

One example Teach gives of this is psychologists: they seem to be people of authority, so they can force you to do things and relieve you of the obligation to act. But they don’t actually have any coercive power they can use to hurt you. He has a nice phrase here, “people prefer power to be on the other side of the glass” (imagine a setting with a psychologist observing you from behind a one-way mirror). They have no real power over you, but complete illusory power. No danger, maximum excuse for inaction.

I see a lot of this in the tech industry. People put themselves in fairly evil positions: optimising addictiveness of youtube shorts to children to increase revenue. Building AGI even though they acknowledge it might kill everyone. Enabling widespread scams in order to collect ad revenue. Yet many workers who build these systems for the tech industry are not sociopaths getting what they can for themselves. They are nerds who are personally quite kind.

If you ask them why they do this, they outsource responsibility to the companies they choose to work for. “I’m just doing a job, they would do it anyways, competitor is worse.” A Teachian reading of this is that they found a petty tyrant, and cheerfully signed up to avoid responsibility for what they do.

There is something similar in politics. For many people, it is better to live under politicians you can blame then do politics yourself. Teach argues that a big appeal of authoritarianism is this abdication of responsibility. We all know that of course direct democracy where everyone votes on every issue won’t work. Teach argues there is a bigger reason we don’t want direct democracy. Direct democracy would mean we are directly responsible for all decisions our government (read: us) makes, which would be terrifying.

In short, if you can find some oppressor to constrain your actions, you have the perfect excuse to do nothing.

Conclusion

Being unable to act independently is so common it is even a major troupe in our fiction.

The standard hero’s journey has a call to action, which obligates the hero to act. It could be a mentor, an authority figure, or some mystic chosen one prophecy. But the key point was that action didn’t come from them and their desires.

The only people acting on their desires in these fictions are villains.

The equilibrium here is quite bad: if good people believe they must be obliged to act, they will mostly not act. The only people acting will be sociopaths and psychopaths who do not care for social norms.

Sociopaths and psychopaths acting while most people do nothing is a very bad place to be. Worse yet, most people will seek out these psychopaths and sociopaths to absolve them from responsibility.

Believing most people actively prevent themselves from acting is quite dark. “People look for reasons to not do anything, from rationalisation to finding people to oppress them” is a very cynical model.

Yet it is also one I’ve started to see a lot after I learned about it, especially when I ask people to do something about things they are complaining about.

One of the most common is a friend or an acquaintance will be ranting about politics. If I say something along the lines of, “Do something, then!”, I now highly expect a certain kind of set of well-rehearsed excuses about why that is impossible.

As dark as these models of human behaviour are, once we’ve diagnosed the problem, we may be able to do something about it.

Exactly what to do, and yet more dark psychoanalytic models will be the theme of future essays.

This is a Guest Article written by Daniel Clothiaux (remember).

It explains a few concepts introduced in the book “Sadly, Porn”. The book is not well-written, but it deals with a dark type of psychoanalysis with which only a few authors dare engaging.

My friend Daniel decided to review it, in order to distill its essence and package it in a way that is approachable by others.

This is very different from my usual articles, most notably in register, it is much more vulgar than the usual Cognition Cafe article.

One of the critical points the author Edward Teach makes in the psychoanalytic book Sadly, Porn is that we struggle to have positive desires. We struggle to have positive fantasies for ourselves and our futures.

Desires are scary. Desires mean we want things, things we don’t have right now. Unfulfilled desires feel bad in the moment, as we haven’t succeeded in fulfilling them yet. Failing to achieve our desires is painful.

To avoid pain and feeling bad, it is easier to replace positive desires with porn (other people’s desires and fantasies) and envy (wanting to deprive people of what they have). Yet not having our own positive desires also feels bad; who wants to admit they are an envious degenerate? To avoid feeling bad, we hide our lack of desire, degeneracy and envy from ourselves, using various forms of misdirection.

Teach presents this set of models in completely universal tones. We don’t have desires, and that includes you. I think it’s better to understand these as common patterns people (including you and I) follow, at least some of the time.

Teach also expresses this (and everything else in Sadly, Porn) very vulgarly. I will be vulgar, here too. I think it is an important element to better convey the vibe of his models. But unlike the book, there won’t be lengthy descriptions of sexual acts in this essay.

Desires

Like I said, having desires, things that you want, is scary. You could fail to achieve your desires, which would feel bad. You could persistently be failing, which would feel even worse.

One easy trick to avoid feeling bad about failing is to never want anything at all. So many people often do not want anything. We suppress our desires so we cannot fail. To avoid failing to achieve our dreams, it is easy to not have dreams and fantasies at all.

There are many, many ways this avoidance mechanism instantiates itself.

Teach often presents vulgar examples. Asking a girl (or guy) out is scary; you might fail. Fucking someone is hard work. Easier to jerk off in your room.

Starting a company is scary. You could fail at many, many points along the way. You could fail to get funding, build a product, find customers, monetise, or scale your product. Much easier not to want to pursue becoming wealthy in the first place.

Publishing blog posts and book reviews is scary. Maybe nobody will read them. Maybe someone will come by and tell you your writing is shit and call you mean names. When I started writing, it was obvious that my writing was (and is) bad. Much easier to just lurk and never want to become an established writer.

In general, much easier to avoid the pain of wanting things by wanting nothing at all.

Porn

If we try to avoid desires, what do we do instead?

One easy way to avoid having desires is to replace them with facsimiles of other people’s desires.

If you jerk off fantasising about yourself fucking a hot girl, this can become a goal of yours. Worse, it is a goal you are currently failing at, as you aren’t fucking a hot girl.

Instead, you watch porn, which is someone else fucking a hot girl. Rather than taking that as a launch pad to your own fantasy (imagining yourself fucking said girl), it replaces it entirely.

According to Teach, porn in a more general sense is something you consume, in order to replace the work of having a fantasy or having a desire yourself with somebody else’s fantasy about somebody else. It’s not even your own fantasy. Because it isn’t your own fantasy, you will never act on it. Do enough porn, and real things start feeling unreal and porn real.

Teach emphasises over and over again that this is cuckold porn, specifically. You are replacing your fantasy of fucking a girl with someone else’s fantasy of someone else fucking a girl. This is pretty literal cuckoldry. Contrast this with a cuckold fantasy where you imagine yourself watching someone fuck a girl; that would be a cuckold fantasy but not cuckold porn.

Many examples he gives are sexual, but not all of them. A lot of modern fiction is similarly metaphorical (cuckold) porn. A standard hero’s journey fantasy story replaces the desire to take action and become a hero yourself with a story of someone else becoming a hero. People love cheering on others as heroes and would often rather replace fantasies of being heroes themselves with the fantasy of cheering on a great hero.

This pornographised replacement is much easier than having desires. It is easier to cheer for your sports team than to fantasise about being the star player. If you fantasise, it will hurt because you are not, in fact, the star player. Even desire is easier than the painful training it takes to become a star.

At this point, everyone who is creating porn also do not have desires, as they also have completely pornified brains. So you get your desires from society and the media, which get their desires from society and the media. It’s copies of copies, none of which are anyone’s internalised desires.

Envy

Another way to easily avoid desires is to feel envy instead. It is often easier to deprive someone else of something than to get it yourself. It is easier to want to destroy things than to build them. This negative emotion is envy, not jealousy. Jealousy is wanting what they have for yourself, but it is harder because you would then need to want something.

Many people want to be desired, but becoming super desirable yourself is hard work, and you could fail. Instead, better to be envious of other people for being desirable. Teach gives a very crude example of a husband envying his wife for being desirable. He could want to become desirable himself, but that is harder than envying his wife.

Resentment and envy toward successful people are very common. Many cultural tropes touch on them. The nerd who envies the jock, his easy popularity and success with women. Ugly women who envy beautiful women and how much easier they have it. Money is another common area where people envy the successful. Rather than make a billion dollars yourself, it is much easier to feel envy towards billionaires and want to deprive them of what they have.

Envy leads us to try to deprive others, and often in very lazy ways. We’d deprive the billionaires of their wealth if we could. Actually depriving billionaires of their wealth is a hard political goal. If we can’t do it, there is another step many people take: imagining they must be miserable. This lets you feel vicious envy without having to do any work. The husband could actively make his wife less desirable, but that is hard work. Easier to deprive her of that desirability indirectly, say, by watching porn instead of fucking her. That lets him make her feel unwanted as lazily as possible.

Misdirection

But admitting you are a degenerate, envious creature addicted to porn would be very painful. Most people lack the courage to look at this directly.

Instead, we bury our lack of desires under layer upon layer of misdirection. We then seek ways to validate our misdirections. Teach basically sees much of psychology from Daniel in the Bible to Freud and modern psychologists as giving people excuses to hide their true desires.

The example of Daniel he gives is one of the clearer ones, though a bit insane. His reading of Daniel is that a king wanted his kingdom to be his forever. He would rather see it ruined than pass it to some inadequate child of his. He couldn’t admit that, and a prophet named Daniel knew that. So he gave him cover by saying that if his kingdom falls, it will fall to the kingdom of God or because God wills it. The king could cheerfully let his kingdom decline out of spite and not feel bad about wanting to deprive his successor.

I’m not sure how real his version of the biblical story is, but there are many more grounded examples. If you are watching porn instead of having your own fantasies and trying to go out and fuck hot people, that would suck to admit. Instead, you will tell yourself “oh, I have a porn addiction” or “I am too tired”, not “I am afraid of desires”. If you envy your spouse and try to deprive them, you will tell yourself, “I’m not fucking them because I’m watching porn because of addiction”, and not “I envy them and want to deprive them of feeling wanted”.

If you post on social media to make other people feel envious of you, you’ll frame it as being addicted to social media. If I am envious of someone with more money than I have, I’ll frame taking their wealth as promoting equality. When I state something that could be a desire, I’ll wrap it in detached irony so it is clear that I don’t actually mean it. Then if I fail, I didn’t want it anyway!

If you envy and despise rich people, your justifications will be “fairness”, “equality”, “the rich are all evil”. If you envy hot, fit people, you will tell yourself, “I am not that shallow”.

All of this to avoid seeing our internal motivations, because that would be painful.

Conclusion

These models are quite ugly.

I didn’t justify them beyond a handful of examples or try to built mechanistic models of them.

Yet I personally have found these concepts useful. To varying degrees, they have helped me understand what happened in my life, both in myself and in the people around me.

I have struggled to articulate and pursue my desires, and I’ve seen many people struggle to tell me theirs.

I see people consumed by media teach would consider porn, and lots of vicious envy.

Not having desires is very bad. Teach touches on some of the downstream consequences of lacking desires.

I will deal with this, and what exactly we should do with this (and the other dark models from Sadly, Porn) in other essays.

With that, as Gabe would finish his essays, cheers!

Knowing things is hard, sharing our knowledge accurately is even harder, and it is all too easy to claim more knowledge than we have.

Many, when confronted with this, fall back to Epistemic Humility. They err towards being under-confident.

This is bad. I’ll explain why here, and then point to an alternative virtue to cultivate instead: Epistemic Precision.

But first, let me give some context.

Social Bids

Before getting to Epistemic Humility, I should introduce the concept of Social Bids.

In the social world, we are all constantly making bids. And the impact of Epistemic Humility is better understood through them than traditional philosophical epistemology.

—

For instance, we make bids for others to see things our way.

In a meeting, when a colleague speaks and audibly says “Obviously, this is a bad plan”, they are doing more than sharing their point of view with the rest of the team.

They are making a bid for the team to accept that this is a bad plan. They are challenging people to contest their statement. If no one challenges them and people seem at least neutral to the statement, the bid will have been accepted.

Once the bid is accepted, anyone in the team can build upon the assertion that the plan in question is bad. As more people do so, it becomes more and more common knowledge.

—

Bids come in all shapes and sizes.

Parents and bosses make explicit and straightforward bids when they order their respective children and employees.

We make bids to consider a proposition, to adopt a frame, or to use specific words.

We make bids, we counter people’s bids, we negotiate and compromise, we accept and reject the bids of others.

We bid using power, authority, status, emotional appeals, and to a much lesser extent, appeals to reason.

It’s the full chaos of the social world.

—

Most importantly, we have a Bid Budget.

Regardless of the type of bid, everyone knows and feels that they have a limited budget.

A parent can only order their children to do so many things before they revolt. The same is true for a manager or a captain.

Similarly, in a group, we can only make so many surprising statements before people start doubting us and in the end stop believing us.

Budgets are hard to manage. And bid budgets are no exception.

I have personally found that most people I care about under-use their bid budgets. They do not negotiate at work, they do not state their wants enough, they do not push their ideas, they do not claim the attention of the group they belong to when they should, etc.

Bids and Epistemic Humility

The impact of Epistemic Humility is better understood through Bids.

After all, from a pure epistemic standpoint, generic Epistemic Humility states to be less confident in everything. It’s almost void of meaning: it doesn’t change one’s beliefs in the end.

But Bids bypass traditional epistemic considerations.

Traditional Epistemology is about considering whether a statement is true or false, whether a plan is good or bad, whether an argument is sound or not.

Bids are largely about selecting which statements, plans and arguments are up for consideration in the first place.

This is crucial in the context of common knowledge: Plan A may be better than Plan B, but if Plan B is discussed more often, it is easier for people to coordinate around it.

For instance, I think of my current situation as an intellectual in these terms.
On the idea front, I believe I am doing well. I consistently get complimented on them.
However, I am failing to acquire enough social capital to bid for them.
This has very little to do with epistemic considerations, and much more with my (lacking) skills at earning social capital.1

—

The effect of generic Epistemic Humility is to weaken its followers.

The generic stance of Epistemic Humility dictates that we should be less confident in our beliefs because of our biases and epistemic failures.

At the very least, it recommends that we should express less confidence and qualify our sentences.

In practice, this is bad.

Sometimes, nerds say things like “I’m not sure but possibly [X]”, “Plausibly, [X]”, or “I think that [X] may be true”.
When nerds say this, they mean to make a regular-sized bid to add “[X] has 20% chance of being true” to what the group believes.
However, groups (not necessarily individuals, but the groups themselves), will interpret it as them making a much-smaller-than-usual bid for the group to believe [X].2

Put differently, the main effect of generic Epistemic Humility is for its followers to behave in a way that lowers their social impact. It doesn’t even make them more truthful, as it doesn’t help the groups they talk calibrate on better probabilities.

—

The effect of selective Epistemic Humility is pernicious.

Selective Epistemic Humility dictates that we should be less confident in some beliefs.

There are two cases in which this may happen.

First is when it is actually warranted. Someone provided evidence that a specific belief was wrong. In that case, it is not epistemic humility at all to believe less in it. It is nothing more than changing one’s mind as a reaction to new information.

When it is not warranted is where it shines. It is an isolated demand for rigour (Scott Alexander on the topic) masquerading as an emotional appeal to humility.

De facto, the selection of beliefs covered by the selective Epistemic Humility will gather less attention than others. It is a sneaky way to reduce their impact, without having to ever make the case that they are actually wrong.

Epistemic Cowardice

I tend to think of generic Epistemic Humility as Epistemic Cowardice.

Epistemic Humility often amounts to taking decisions to protect one’s ego from the risks associated with uncertainty. Risks of being wrong, of people showing that we’re wrong, of being ridiculed for our beliefs.

In practice, I see people being ill-at-ease when they are uncertain. So, in a manner not too dissimilar to cognitive dissonance, they try to reduce their uncertainty by staying in their comfort zones.

Instead of Epistemic Humility, I find that I need people to be much more Epistemically Brave. I need them to figure out what’s best, and to be at ease with committing to courses of action even in the presence of uncertainty.

They should not quash their feelings of uncertainty and assume they are right, instead they should stay the course and not quiver in spite of the uncertainty.

—

Institutions and politics require a lot of Epistemic Bravery to go well.

From my point of view, I see that both are lacking in qualified smart people.

In my experience, Epistemic Humility has been a direct cause of this.

Many smart people told me things like:

I’ll never engage with politics. It’s so corrupt, so hard to figure things out, and so easy to cause things to go wrong.

Only psychopaths or power-hungry people are willing to do this, and I am neither.

And as a direct result of these thoughts and feelings, they have made their decision to not engage with institutions and politics.

At best, this is Epistemic Humility gone awry, wherein people feel that they do not deserve to Take Power, even through the Rightful Means.

At worst, it is a cowardice finding a convenient excuse in Epistemic Humility.

—

The same problem exists in situations with smaller stakes.

In many social situations (think of families, friend groups, corporate offices), when things get hairy, there are a lot of people who go “Oh, I don’t want to take sides in this drama.”

This attitude strongly empowers sociopaths. They know that until they screw up badly, they have a clear field to abuse people.

For the same reason, their victims are alone and have to fend for themselves. They have to bring neatly packaged irrefutable evidence to the group before it acts, which is very hard to do without the support of the group.

Victims often even get blamed when they try to prove what is happening! While the group doesn’t know who’s right, the victim’s attempts to gather proofs will be perceived as accusatory, mean and paranoid.

Sociopaths are aware of it. And they leverage it.

Victims are aware of it. They often resign themselves as a result of knowing people won’t help them.

This is morally bad.

I don’t mean that one ought to get involved with all the drama that happens around them: we are puny little humans. We have a limited ability and will to do what’s Good, and we thus necessarily do bad things.

But I think one should still be aware when we are doing something bad. We should especially not elevate our weakness to a virtue. We should not act as if we were superior to the people going through drama by virtue of ignoring it.

—

A friend responded to the previous section with:

I agree this is a real problem! Often, though, I think it arises out of avoiding acting out of fears of the consequence rather than any stance of Epistemic Humility.

This may be an excuse, I’ve more often heard people directly state they didn’t want to make anyone mad in these situations, or they prefer to reserve their role as a peace keeper, or so on (and have done so myself, for better and worse).

I believe my friend is largely right, but under-appreciates the extent to which ensuring that good norms are followed takes both common knowledge and someone getting dirty.

These two things trade-off: the clearer a rule violation is, the easier it is to punish the violator. And conversely, as a situation gets less and less clear, the more the rule-enforcer will have to bid in order to investigate and fix it.

When someone violates a good rule, someone else must punish them, and it takes a toll. At the very least, they must go to the rule-breaker and tell them “Hey man, you broke the rule.”

The rule-breaker will always counter-argue. Either because they acted in bad faith, or because they acted in good faith and feel the need to justify themselves.

Thus the rule-enforcer will have to pay a social cost to make the bids needed to make it clear that nope, the rule was broken. They will be the one asking people to pay attention to the arguments of the different parties, and take on the role of both prosecutor and executioner.

Then, the rule-enforcer will additionally need to pay the social costs required to enforce an eventual punishment: whether it is an apology to the group, a promise to not repeat the behaviour, a penalty, an exclusion, etc.

And finally, they will be the one to incur the dislike (or the wrath!) of the punished party.

This can all be mitigated by common knowledge and clarity. The more common knowledge there is about what happened and what the rules are, the smoother all of this goes. The less it costs to the person who will enforce the rules.

—

“Oh, but, can we really know what’s good or bad? How can we ever figure out who’s right or wrong? Can we really know if they meant to act badly?”

Such undirected Epistemic Humility only serves to weaken common knowledge. It thus becomes much harder for someone of comparable status to fault someone else.

This is not hypothetical.

Join spaces that take pride in their open-mindedness and humility, and you will see a lot of enablement of sociopaths borne out of “Oh, but who is to know what is truly bad or not???” to any bad behaviour on the edge.

In practice, enforcement in such places is even more asymmetrical than usual. It only happens when a person of low status hurts a person of high status.

This is in stark contrast with a Rule of Law, where Laws are Respected, and where everyone is equal in front of the norms.

Epistemic Precision

Now, onto a more positive vision. In fact, I have an alternative to Epistemic Humility.

Instead of Epistemic Humility, I recommend thinking in terms of Epistemic Precision. Epistemic Precision is not about being humble about what we know, but being precise.

We are not random machines, outputting random sounds and writing random symbols.

There’s always a reason for why we think what we think, say what we say, and do what we do. Epistemic Precision is the practice of paying close attention to it, enough to get a reliable understanding of where one’s confidence comes from.

Let’s go through some examples.

Benchmark

I was once asked for feedback on a benchmark suite. It claimed to measure [some property].

But the author did not think that their suite was in fact measuring the property. They didn’t think the benchmarks were close to doing so, and they never used the suite themselves to evaluate [the property].

I thought this was thus blatant academic misrepresentation, and told the author I thought he was lying and should not do so.

—

They thanked me for reminding them to be Epistemically Humble, and to instead claim that their work was only a stepping stone to measuring the property.

I vehemently disagreed!

The assertion “The suite measures [the property]” is wrong. 10% of it is still wrong, just 10% as much. The direction was incorrect.

So they asked me if I thought they should retract. They were sad about it, because they put in a lot of work, and they thought it could still be useful.

I asked them why they thought it could be useful. They responded that they crawled through hundreds of benchmarks to build the suite, and that even though the benchmarks were bad, these were the closest they could find.

And my conclusion was: “Just say that!”

Indeed, “just saying that”, saying what they believed, would have transformed the project. It would have gone from “one more example of Academic Misrepresentation” to “Providing strong evidence for one of the major problems of the field: that its objective measures are thoroughly inadequate.”3

This is the magic of Epistemic Precision.

Soft Knowledge

The example above was a bit too academic.

Where I have found that Epistemic Precision shines the most, is in situations that have to do with less cut-and-dry knowledge.

Here are a few small examples.

—

Quite often, I ask a question that seems deceptively easy. Not on purpose, it’s just how the world is: sometimes, questions seem easy to answer but they are not.

In these situations, I often get an unreflected answer, that reflects more their preconceptions than their actual thoughts.

I then follow up with “Do you know the answer to be true, or are you inferring that it is?”

It helps them a lot with pausing, and realising that they were making too many assumptions. Assumptions that I may not know about, that I may disagree with, that they themselves may disagree with, or which may turn out to be wrong.

—

On another note. As a human being, a lot of my knowledge is intuitive.

I value my intuitive knowledge quite a lot. But it is sadly much more subtle than formal knowledge, and thus hard to capture with words.

I also value other people’s intuitive knowledge. Sometimes, someone will make a claim that doesn’t seem natural to me. When I try to tease out why they make that claim, they become defensive.

Instead of an “I got this intuition from [doing X]”, they often start a barrage of rationalisations and fake post-hoc pseudo-logical explanations.

When I tell people about said rationalisations (by engaging with them, by telling them I want to learn about their intuitions, etc.), they revert back to some Epistemic Humility: “Oh well, I guess I don’t really know…”

This is so inefficient! There is a reason why they hold their intuitions and paid attention to them, it’s not just random noise. By paying attention to it, introspecting and understanding where their intuition comes from; we can then both learn and infer even more than what they immediately intuit.

Instead of them being Humble, I want them to be Precise. I want them to tell me what their intuition feels like, what reinforces it, where they have seen it work well, etc. I don’t want a bland “Welp, I guess it’s not Proper Formal Knowledge and it is thus Worthless, I should be Epistemically Humble and ignore it :(“

Indirect Knowledge

I have found Epistemic Precision to also be very valuable in dealing with “Indirect Knowledge”. Indirect Knowledge is knowledge that I have not gotten by myself, but instead that I have gotten from books, people, social media, and the like.

To me, “Indirect Knowledge” doesn’t feel real, it doesn’t feel concrete.

—

For instance, when I think “The sky is blue”, I have a pretty clear impression of what I feel and expect. I can think of all the time I have seen the sky being blue, I know what I mean. It is real knowledge.

When I think “The sky is blue because of scattering”, it feels super fake.

I can of course try to come up with an explanation. I have learnt some optics and some wave physics, but man.

Let’s consider the first explanation I come up with: “The atmosphere is mostly made of nitrogen. And nitrogen scatters blue more easily than the other colours.”

Even though I said that, I don’t know why fog makes everything grey (“water scatters grey more?? it’s not even in the light spectrum!”); I don’t know why the sky is orange during twilights; I don’t know why it is purple during typhoons. And to be clear, I haven’t played much with gaseous nitrogen either.

This is why “The sky is blue because of scattering” doesn’t mean much to me.

—

If I ever read “A paper shows that liberal christians are kinder than conservative atheists”, I wouldn’t even perceive it as “knowledge”. Like, this will not change, not at all, how I perceive liberals, christians, conservatives and atheists.

The only knowledge that I would have gotten there is that a “researcher” wrote it, and managed to pass it as “science” to many people.

To a large extent, this is how I relate to most of my indirect knowledge.

At this point, whenever I read claims that are not tied to a specific operationalisation, I treat them as social claims. A form of emotivism.

Concretely, if I came to read the headline above today, it would purely register as “Hurray Liberal Christians! Boo Conservative Atheists!”

—

To be clear, there are things I do to internalise indirect knowledge.

For instance, I initially learnt about solid dynamics and mechanics from books.

But through many exercises, watching standard experiments, building things myself, talking to teachers about it, and more; I managed to make this knowledge mine.

When I talk about solid dynamics and mechanics now, I know what I mean. I can dig into the details.

I now consider it direct knowledge.

—

However, if I am asked about something that I only know indirectly, I will be clear to the person about the fact that the knowledge I am relaying is indirect.

If I have a good recollection of where I got it from, I’ll say something like “Word is on social media that […]”, “I have read in a book that […]”, “Some researcher wrote in a paper that […]”.

If I can’t, I’ll say “I can’t remember where, but I remember having heard/read that […]. I have never experienced it myself though.”

Or, the worst, “I can’t remember at all. But I think someone once said [X] or something similar. Given how little I remember about it, I don’t even know what they meant.”

At that point, is it even knowledge? I do not think I would purposefully change my decisions based on this, nor that any of my interlocutors ever would.

This is how I deal with indirect knowledge in the framework of Epistemic Precision. I have cultivated an ability to state what nature of knowledge I have, how indirect it is, rather than just a vague feeling of “I should be humble.”

Conclusion

Armed with all of these concepts, the core points of this essay can be neatly summarised.

1) Epistemic Humility doesn’t serve an epistemic function. Its main effect is to weaken one’s social bids.

2) It is thus a good fit in situations where people benefit from weakening their own bids and not standing for their beliefs. This makes it a convenient cover for Epistemic Cowardice.

3) Instead, the actual virtue is Epistemic Precision. Being clear and confident in one’s actual beliefs. While these beliefs may be intuitive or indirect, one should still be clear about them.

On this, cheers!

We learn and we forget.
We get addicted and traumatised.
We build habits and lose them.
We discover new facets of reality, and start ignoring them.
Our personality changes. We change.

The question of how people change is complex. But it is critical for understanding the world, how it shapes us, and how we shape ourselves.

This question is among the most important ones in psychology. It underpins memory, trauma, our sense of self-worth, our relations to others, AI psychosis, and so much more.

—

Paradoxically, despite how pervasive it is, there is no name for this phenomenon.

For the change we go through as a result of experiencing something.
There are more specific words, like “conditioning” or “learning”.
There are more generic ones, like “change” and “transformation”.

But there is none for the actual thing. So I will arbitrarily pick one: Human Fine-Tuning”.

Before analysing Human Fine-Tuning in depth, let’s start with a few examples.

A Few Examples

Vocabulary List

Sometimes, the changes to our brains are directed and purposeful. In which case we call it learning.

For instance, we set out to learn a vocabulary list in a language in which we hope to become fluent. By doing so, we hope to enact many changes on our brains.

First, we want to learn to understand that new language. More precisely, we want our brain to naturally conjure the relevant concepts when faced with the words.

Second, we want to learn to speak fluently in this language. When we need to express the concepts from the list, we want the words to come naturally. However, this is hard to get just from working on a vocabulary list. So, at the very least…

Third, we want to keep the list of words in our memory. That way, when we will need to express the relevant concepts, we will be able to think hard about them (instead of having the words come naturally), recall the relevant words, and construct our sentences with a bit of effort.
All of this, knowing that the more we practice, the more fluent we’ll get.

But the changes do not stop there.

Fourth, we develop familiarity with the language.
We get a feeling of its etymology: does the language mostly come from Greek, Latin, Chinese or Arabic?
We get a feeling of how it sounds, and what it looks like. Does it have an alphabet, or ideograms? Does it have a simple set of sounds, or a large variety of throat consonants?
We get vibes of how the words are constructed. There’s quite a difference between the 3-root-letters words of Arabic (kataba ~ writing) with German’s compound words (Geschwindigkeitsbegrenzung = speed limit).

Even with something as direct and directed as a dumb vocabulary list learnt by heart, there’s a lot to say.

American Diner

However, most changes to our brain are not purposeful and directed.

As I was writing this, I remembered a fun anecdote.

When I was younger, I had seen many American diners in movies – or TV Shows, it’s hard to remember and that’s kind-of the point.

I never thought much about these diners. I’d see them, largely ignore them, and focus on the plot instead.

I hadn’t even learnt the word “diner”. As a Frenchman, and because of their ever-present context, I simply assumed it referred to a special type of restaurant (which it did!), never paying much attention to it.

But nevertheless, in the background, a lot happened.

Even though I never paid the word “diner” much attention, I had a feeling the US would be filled with these recognisable restaurants: pancakes, coffee, nice waitresses, cozy booths with their red-vinyl benches, a counter with its typical wooden stools.

Coincidentally, 10 years ago, a friend invited me to a French “diner”. Or let’s say, a pale imitation of one. It was much too clean! The red vinyl was not cracked: it was shiny. It didn’t feel cozy at all, it was artificial, the music was slightly too loud, and the neon lights were a bit too kitsch.

I didn’t think much of it back then. But reflecting on it, it is actually quite impressive.

I had built an opinionated aesthetic sense of a thing that I had never experienced myself. That I had never even named.

Just from seeing them from time to time in movies, I came to associate them with certain attributes, certain feelings. And visiting the one in France; it felt dissonant. Or more than dissonant, it felt wrong.

I don’t think there was a big conspiracy, where Big Diner was trying to sell me more Diner, where diner chains lobbied all of Hollywood to systematically feature them in movies and show specific qualities.

It just happened. The aesthetics of a French kid fed on Hollywood movies was moulded in a meaningless way. That’s just the way the world and our brains work.

But it happens to everyone, constantly. Simply by exposing ourselves to pieces of art and media, we build strong opinions about everything. Said opinions inform our experience of the world and thus our actions, without us noticing that we even formed them.

Loss

So far, I have been pointing at minor changes. But sometimes, these changes can be big.

Like most people who have the chance to live long enough and build meaningful relationships, I experienced loss a few times.

My latest loss experience hit close to home, was particularly violent, and had a sizeable blast radius.

Loss hurts everyone, both in similar and different ways.

But what personally hurt me was having to witness people close to me lose a part of themselves. Each of them had been durably changed, and for the worse.

A visible hole had been carved in their soul. I can see the sadness through their eyes whenever a topic directly reminds them of the loss. They visibly carry more weight: they stand less straight, they are more tired, and they are less optimistic.

It is tragic. Violent loss is of one of these few experiences that make people into a durably worse version of themselves.

Why am I writing about this? Not to make you sad. I promise there is an actual point.

—

The point is that young enough, I had noticed that adults looked like they were missing a bunch of obvious things.

They had lived their entire lives without learning a facet of engineering and building things, without ever pursuing an art form and creating, without trying to get into politics.

When discussing and debating, they would miss obvious arguments, and would get angry when I’d try to correct them.

They were missing so much. Experiences, lines of reasonings, courses of actions; which all seemed obviously important to me. It felt like adults were dumb, for no good reason, and in a way that resisted me trying to help them.

Over time, I figured out what was happening. It’s not that they were dumb and missing the obvious things. It’s that they were explicitly avoiding them. These things made them feel bad.
They knew their artistic pursuit would be a struggle, they knew they were likely to fail any ambitious political endeavour, and they wanted to avoid that.

Later, I learnt about the word trauma in the context of PTSD.
Even later, I learnt its more generalised meaning of emotional damage.
This made it easier to communicate the observation from above.

People get traumatised. As a result, they become behaviourally stupider versions of themselves, in a way that resists mending.

From my point of view, people accumulate chip damage over time. And ultimately, they die of a thousand cuts. They are too damaged to willingly try new things and put themselves out there.

This has been one of the sadder parts of my life. Seeing people slowly lose Their Spark as they internalise all the bad things that happen around them.

Mechanical Analysis

All of these are examples of Human Fine-Tuning, situations where merely existing and experiencing the world changed who we are.

These situations are all different. Some are happenstance, and others are purposefully directed. Some are purely logical word-level associations, and others are deep changes to who we are.

More often than not though, we naturally mould ourselves into what we perceive.

This general process of “a brain changing” doesn’t really have a name. So I am going to apply to people the closest term that I know: Human Fine-Tuning (HFT).1

As Wikipedia puts it:

Fine-tuning involves applying additional training (e.g., on new data) to the parameters of a neural network that have been pre-trained.

Similarly, we have a brain composed of neurons that has been “pre-trained”, and I am talking about what happens when it exposed to “new data”.

Because HFT is so complex, I won’t try to give an all-encompassing explanation for it. Instead, I’ll go through 4 different high-level mechanisms.
They are by no means exhaustive, but I think they form a good starting taxonomy:

1. Associations. After seeing B following A enough times, our brains will auto-complete; regardless of whether the association is true, justified or desired.

2. Aesthetics. Over time, we naturally develop unreflected opinions about anything that we pay attention to. We often mistake them for endorsed judgments.

3. The Audience. We imagine the reaction of people whose approval we seek. Experience changes which people these are, and how we imagine them.

4. Ontological Nudges. Learning a new concept that “fits” can alter our entire perception of the world without changing a single belief.

1) Association

Associations are the simplest mechanism. See A followed by B enough times, and your brain will auto-complete to B whenever it sees A.

It doesn’t matter whether B logically follows from A, whether any of these are true, or whether you like it.

The quintessential version of this is Deez Nuts. Whenever a friend ends a sentence on the word “these”, say “Nuts”. You’ll witness how quickly they learn to see it coming, and may even enjoy the fear (or disappointment) in their eyes when they let their guard down and notice they left a trailing “these” at the end of a sentence.

French is filled with these.2 “Quoi? Feur.” “C’est qui qui a fait ça ? C’est kiki !” “Hein ? Deux.”

—

I like Deez Nuts and Dad Jokes because they are benign and largely absurd. No one is deceived by them.

Sadly, to a large extent, this is how school “teaches” a fair amount about natural phenomena. “Why is the Sky Blue?” → “Scattering”.

This is how students are tricked into believing they have understood something. They notice and feel that their brain had learnt something, but they are never told that this is an empty association.

Idiocracy makes fun of this phenomenon. In the movie, people are watering crops with a sports drink (Brawndo). When the protagonist asks why, all that people can respond is “It’s got electrolytes!”, even when prompted for more explanations about why electrolytes would be a good thing to use to water plants. “Why is Brawndo good?” → “It’s got electrolytes!”

Ironically, real-life makes fun of Idiocracy. Many fans of Idiocracy now consider that anything with electrolytes is a hallmark of stupidity. Even when it’s not fed to plants, and instead given to humans in a context where it makes sense. They have learnt the association “It’s got electrolytes” → “It’s stupid!”, and do not realise that it is empty. This is how we end up with such threads.

—

Back to Deez Nuts. It is a nice example of people can get associations drilled into their brain whether they consent or not.

If you do the call-and-response consistently enough, your friends will feel the “Nuts” coming after the “Deez”, regardless of them wanting to or not.

Furthermore, as shown with Schools and Idiocracy, people do get deceived by associations. They don’t naturally notice how empty they are.

One may wonder. Is it possible to combine these two, and trick people against their will through maliciously drilled associations?

The answer is “Of course. And people do so constantly.” Much of rhetorics, memes and art is built precisely on this.

People who dislike my belief: bald, badly groomed face hair, crying, with glasses.
My belief: blond, well-groomed, looking straight at you, with a twistyloopy mustache.

My opinion: Standing, buffed, dogecoin.
Your opinion: Sitting, crying, sad doge.

—

In real-life, the memes are rarely that straightforward.

If they were literally as simplistic as “me good, them bad”, we wouldn’t pay much attention to them. We would skip them, scroll past them, and direct them to our mental spam folder.

So instead, they feature a level of indirection. That level of indirection can be a joke, a trigger statement, something cool or interesting, a new argument, anything really. Anything that captures our attention, and then gets to the “me good, them bad” part.

That is all that is needed for the fine-tuning to happen. Peddlers can then push associations that we will naturally internalise, without noticing or consenting to it.

—

Associations are straightforward and very salient. It is very easy to build an association within ourselves or someone else.

But not all HFT is about inferring and completing patterns.

2) Aesthetics

Aesthetics are more subtle than associations.

“Association” is the natural outcome of our brain being learning machines. Brains love learning.

“Aesthetics” is the natural outcome of our brain being value machines. Brains love judging.

This XKCD comic explains it in a funny way.

Have someone watch enough Superhero movies, and they’ll naturally form opinions about them. They may feel strongly about them. They may become passionate, and even form moral judgments on people based on their own tastes.

Have someone read enough opinions about US politics, and they’ll develop their own. Even if they’re not from the US.

This effect is pernicious. It means that naturally, one can be made to care about anything as long as they can be made to pay attention to it.

And this can get worse over time. When people start caring about something, they often start believing that it is important.

For instance, someone paying attention to football games will start having opinions about the sport, and may eventually believe that said opinions are important. Same thing for Reality TV, video games, nerd lore, etc.

—

But the brain-hack doesn’t stop there.

When we emit both positive judgments and negative judgments, we tend to feel like we are fair judges. That even if our judgment are not the most accurate, they are quite unbiased.

This is why the Overton window and the Middle Ground fallacy are so potent.

Let’s say that someone is only ever exposed to rightist opinions. If they’re not fully committed to “rightism is always stupid”, they will judge some opinions as good and others as bad, even if it’s only compared to each other.

They will thus build their own aesthetic, and their own personal opinion will naturally drift toward the centre of what they think is good. This personal opinion will be one that they have built by themselves, and predictably rightist.

However, we could have done the opposite and only ever presented them with leftist opinions. In that case, their own personal opinion would have been a leftist one!

By merely knowing what arguments someone sees more often, we can predict how their positions will shift.

—

This also explains the Fundamental Attribution Error.

People tend to think of themselves as – if not perfect – fair reasoners.

Let’s say we have Alice, Bob and Charlie. Alice is conflict avoidant, Bob is normal, and Charlie is aggressive.

From each of their points of view, their own level of conflict is fair.

Alice doesn’t literally always say YES to people. If she did, she’d be a proper slave, fulfilling everyone’s desires.
Of course, she has her own criteria for when to say yes or no. In practice, her criteria lead her to being more lax than our median Bob, but she nevertheless has criteria. Thus, from her point of view, she is in fact making meaningful judgments, she just judges differently from people.

Conversely, Charlie doesn’t literally always say NO to people. If he did, he’d be a fully anti-social person and end up in jail.
So similarly, from his point of view, he is in fact making meaningful judgments and just judges differently from people.

Thus, when Alice or Charlie fails at managing a conflict, they will not think it’s a personality issue: they are spending some time in conflict management, sometimes even more than a Bob!

Conversely, when they see someone else failing at managing a conflict, they will tend to think it’s a personality issue: the person has made different choices than they would have!

—

Aesthetics percolate all aspects of the human experience.

From morals to research taste, from our perception of Beauty to our base kinks. Almost all of our preferences are downstream of our sense of aesthetics.

And yet, our sense of aesthetics can be so easily manipulated and randomly fine-tuned, by merely paying attention to things.

Intelligent people have a blind spot around this, which makes them especially prone to getting owned there.

Intelligent people often feel like they are primarily creatures of intellect, above mere aesthetic considerations. Because of this, Aesthetics lies their Shadow. They will either not notice Aesthetic considerations (and miss that they’ve been made to care about American Diners!). Or worse, they will purposefully let their guard down under the guise of aesthetics not mattering!

—

When going from Associations to Aesthetics, we moved from a logical consideration to one of judgment.

Associations can be thought about in objective terms. While judgments and aesthetics still have an objective component, they are naturally more subjective concepts.

This made it harder to write about. But the next topic goes even deeper.

3) The Audience

The Audience is a deeply psychoanalytical concept. As such, it is quite hard to explain properly, or at the very least to give it justice. I’ll try nevertheless.

—

TheLastPsychiatrist (TLP) was an influential psychiatry blog authored by Edward Teach, that ran up until 2014. In it, he often discussed TV shows and movies. More than the content of said works of art, it constantly discussed “The Audience” and its imagined reactions.

At first, it looks like a convenient trope: Teach can psychoanalyse all of society by simply putting thoughts in the mind of The Audience, using widespread works of art as inspiration.

The first level of analysis is simple. Narrative works of art fine-tune The Audience. And TLP describes the process of fine-tuning, as well as its results.

—

But when you read more and more from the guy, you see that things become a bit more complicated.

Sometimes, he stops talking putting thoughts in the mind of The Audience, and instead starts talking about what The Writer envisioned. He tries to answer the question “Why did The Writer write things that way?” to explain why stories are so convoluted.

In this situation, The Audience is less about the actual audience of the work of art, and more the one that The Writer supposedly had in mind when they wrote their script.

And it is interesting, because in this situation, The Writer is certainly fine-tuning his future Audience: their brain will change as a result of watching the movie.

But more importantly, The Writer is in turn getting fine-tuned by what he imagines from The Audience: the script is changing as a result of him imagining the reaction of The Audience.

—

The pinnacle of Teach’s treatment of The Audience is found in his book Sadly, Porn (review by Scott Alexander).3

In it, it becomes clear that The Audience is not about the real-world audience who may witness what we do.

The Audience lives within our minds.

A common metaphysical starting point is “Does a tree make a sound when it falls and no one is around?” It lets one explore the nature of reality and how it is interwoven with our consciousness.

Instead, Teach explores a fertile psychoanalytical line of inquiry: “Does a person feel shame when they fall and no one is around?”

The answer is Yes!

Teach’s answer is The Audience.

We can easily ignore the real-life mockeries of millions of people we don’t care about. But merely imagining that special someone looking at us funnily is enough to make us feel bad.

This is what The Audience is about. This is who it is. Not the special someone, or at least, not the one from the real world. It is the imagined special someone that resides in our mind.

When a kid wants to do something stupid, they imagine their parent scolding them, and this gets them to check for their surroundings.

This is The Audience.

The Jesus in “What Would Jesus Do?”, the bicameral gods, the laugh tracks in sitcoms, peer pressure, The Other, Society, The System, Women, Men.

—

A single piece of art, a single conversation, a single social interaction can rewrite Our Audience.

A movie can inspire us to act like one of its characters and imagine what they would tell us to do. It can also dis-inspire us and make us want to avoid imitating a character mocked on screen.

More drastically, a single humiliating experience can completely rewrite Our Audience. Being Rejected in front of People.

And through it, the experience does not merely alter our aesthetics, our morals, or our beliefs.

It does much, much worse.

It rewrites our social emotions.
Our entire understanding of the social world.
What’s Cool and what’s Cringe.
What’s Pride Worthy and what’s Shameful.
What’s Confidence Boosting and what’s Humiliating.
Who is Authoritative and who is Conspiratorial.
What argument is Elegant and which is Convoluted.

As a wise-man once wrote:

Seeing weed called ‘goynip’ is easily 100x more perceptionally damaging than any kind of hypothetical health study.

—

In the end, I think my treatment of The Audience was not that bad. But I’ll quit the psychoanalysis.

Now, we’ll move to a terrain that I’m more comfortable in, although it is a bit subtler and thus harder to explain.

It has little to do with our associations, judgments or social senses.

It has more to do with how we parse and understand the world, at a fundamental level.

4) Ontological Nudge

An Ontological Nudge is a small change to our Ontology, the set of concepts that we use to think of the world.

Let’s start with a short example.

When I was a young child, I learnt about “nests”. As in, regular bird nests. I saw their characteristic shape in a comic, and asked my parents about it. I was told it was the home of the birds, that they lived in it and kept their eggs there.

It made a strong impression on me. And when I saw one in a tree in the city, I was excited! I learnt about a new element and recognised it in the world.

I grabbed my parents, and pointed at the nest. Then, I was disappointed. No birds came out of the nest. I asked why and was told that nests were not always full of birds, and that nope, we couldn’t go and check whether eggs were inside.

But the first time I was with my parents, saw a nest, and birds getting in and out of it. It was crazy. Boy was I excited.

My Ontology was expanded, and it felt great.

—

In the example above, what’s happening is hard to describe.

Basically, a new concept had been introduced into my brain. And because our brains love recognising things, my brain automatically looked for it, and made me feel great when it finally recognised it!

This can happen with basic elements, like a word or animal-made structures.

Most importantly though, the same happens with more advanced concepts.
Like the Woke “micro-aggressions”.
The Nerd “incentives”.
The Freudian “projections”.
The Consequentialist “utility functions”.

Learning about such general concepts is very pernicious. While they don’t change our beliefs, they change our ontology. The very building blocks that we use to interpret the world.

And they can be changed so innocently. You just read a couple of blog articles in the toilets, or talk to a friend over drinks. You see or hear a word you don’t know about. You check it out online or ask your friend.

Boom, you start recognising it everywhere.

After that, all of your subsequent observations are tainted forever.

—

It doesn’t matter whether “incentives” or “micro-aggressions” exist or not.

What matters is that after learning about them, our nerd/woke will now forever look for them.

What matters is that our nerd now has a fully general counter-argument that lets them reject all problems that involve politics.

“It’s the incentives!”

Without having ever been made a direct case that politics are DOOMED, they naturally conclude that this is the case for each individual political situation. It’s the natural outcome of a nerd having learnt about “incentives”.

They would have rejected a direct case that politics are DOOMED. They are reasonable.

But by changing their ontology, there is nothing to be rejected. Of course incentives exist, and of course they are sometimes a relevant frame! How could you reject that?

Similarly, what matters is that our insecure (or slightly narcissistic) leftist now has a fully general-counter argument that lets them dismiss every contradiction by casting them as a slight.

“It’s a micro-aggression!”

Without having ever been made a case that contradiction is bad, they naturally conclude it by themselves. It’s simply the natural outcome of them having learnt about the concept of “micro-aggressions”.

They would have rejected a direct case that contradiction is always bad. They are reasonable.

But by changing their ontology, there is nothing to be rejected. Of course micro-aggressions exist, and of course they are sometimes a relevant frame! How could you reject that.

—

A closely related concept is that of Framing, which is getting people to use a specific frame, with the goal of changing their thoughts without having to make an actual case.

Ontological Nudges are deeper than a simple frame. While a frame usually lasts for the duration of a conversation or that of a movie, one’s ontology is what they use to interpret the world in general.

Ontological Nudges are also usually smaller than a full frame. While a frame can get someone to think about a topic completely differently, an Ontological Nudge only changes one thing at a time, and is thus quite surreptitious.

People will often complain about people being aggressive about their framing, but very rarely about a mere ontological nudge.

Conclusion

I believe that HFT is a pervasive phenomenon that affects everyone.
It affects you, it affects me, and it affects everyone else.

Internalising how it works is crucial for understanding the world. Furthermore, everyone likes to think they are above that. But no one is.

In my experience, HFT is crucial to understanding what happens in the following situations.

People get converted and de-converted.
Public intellectuals get captured by their audience.
Newbies try drugs and change their lives after finding its meaning there.
Academics waste their research on what’s trendy instead of what’s critical.
Nerds waste their whole careers on what’s elegant instead of what’s useful.
Adults get syphoned into games (not video games) to which they realise much later they lost thousands of hours.
Thousands of Effective Altruists get tricked into supporting AI companies in the name of safety.
Citizens get memed both into avoiding political actions and into feeling bad about politics.
LLM power-users fall prey to AI Psychosis.

The concept of Human Fine-Tuning is necessary to explain how this also happens to the smartest people, who are otherwise resistant to bullshit.

It is at the core of cognitive security and defensive epistemology. I’ll deal with these more meaty topics. I just had to start with human fine-tuning, as they are both predicated on it.

On this, cheers!

Terminal Cynicism is a level of cynicism that leads one to perceive everything negatively, even obviously good things. It is cynicism so extreme that it renders one incapable of productive thought or action.

The most common instance is people refusing to engage with politics in any shape because “The System is corrupt”, thus neglecting it and leaving it to decay.

At a personal level, Terminal Cynicism is dangerous. It feeds on weakness and insecurity, and alienates people.

I also believe that Terminal Cynicism is not always natural and organic. Instead, that it is quite often caused by agitators who spread Fear, Uncertainty and Doubt (FUD).

And defeating FUD requires a lot of clarity. So let’s clarify things.

Examples

First, consider a couple of beliefs commonly intertwined with Terminal Cynicism…

—

“Air Conditioning is bad.”

A plethora of articles explain at length how bad AC is: because of greenhouse gases and cooling fluids, because many poor people don’t have access to it, and how there are actually many great alternatives to it!

But these articles are taking the problem in the wrong direction. AC is good: it makes people’s lives better.

If there’s a problem with how much energy people use, we can tax or ration it. If there’s a problem with the greenhouse emissions coming from people’s energy consumption, we can tax or ration them.

Trying to manage problems that far upstream in the supply chain through individual consumption is more about moralisation than efficiency.

—

“Doing Politics is bad.”

There are much too many smart people who believe that doing politics at all is bad. That all politicians are bad, that wanting to do politics itself is a red flag, and even that it is meaningless to want to do politics as everything is corrupted.

This is harmful and self-defeating. Our institutions rely on people actually doing politics. We need competent politicians, competent citizens, people invested in political parties, and more.

Erasing oneself from politics is the central example of Terminal Cynicism.

—

“Vaccines are bad.”

The anti-vax movement has moved from a fringe conspiracy theory to a widespread belief. It ranges from the belief that vaccines do not work to claims of vaccines containing microchips that interact with 5G towers.

This is wrong. But beyond its wrongness, it is the result of escalating distrust. No honest investigation ends up with “5G microchips” as its conclusion.

The facts of the matter are fairly straightforward. Vaccines as a class have been a powerful tool to erase diseases and slow their spread. While not all vaccines are equally good, they are among the triumphs of medicine.

Let’s be clear: it makes sense to discuss the efficiency and the risk profile of a vaccine. This is why we have long and documented procedures to establish a vaccine as safe and useful.

Similarly, it also makes sense to discuss whether vaccines should be mandatory. This is a non-trivial public health question. Even though mandatory vaccines let us eradicate diseases in the past, it did come at the cost of personal freedom and bodily autonomy.

Overall though, vaccines have been a public health victory. There are many reasonable compromises and trade-offs that are meaningfully debatable. But not whether vaccines contain Bill Gates’ microchips. That one is just Terminal Cynicism.

—

There are many more examples of this type of Terminal Cynicism.

Terminally Cynical beliefs are usually wrong, but not always. It has little to do with the philosophical intricacies of the underlying question or the sometimes subtle truth of the matter.

Instead, what makes it deeply wrong is that it stems from a cynicism so bad that it will take a thing that it itself thinks is good and frame it as bad on purpose.

Here are examples of Terminally Cynical beliefs, along with an example of the cynical justification.

Sometimes, I give two such justifications. I do so when Terminal Cynicism has led to both the typical far-left and far-right clusters to independently build their own cynical justification for why a given virtue is actually bad.

Other times, I give two pairs of beliefs. I do so when Terminal Cynicism prevents either side from looking for balance or a synthesis.

Consider:

“Science” is bad: it’s a white construct demeaning traditional wisdom. “Universities” are bad: they’re an institution that has been fully captured by wokes.

“A Strong Military” is bad: we should never enforce our version of the international order, and let others do so instead. “Opposing Russia” is bad: we should never enforce our version of the international order, and let others do so instead.

“Atheism” is bad: it may be factually correct, but atheists are responsible for the most destructive totalitarian regimes, and atheism doesn’t literally answer all questions. “Religion” is bad: it may help people live better lives, but religious people are responsible for many of the worst moral systems in the modern world, and religion is literally wrong on a bunch of factual questions.

“A Solid Police and Judicial System” is bad: they sometimes make mistakes, which means we should defund them. “A Solid Police and Judicial System” is bad: due process often lets criminals get away, which means we should completely bypass it.

“Hard Work” is bad: it’s a bourgeois fantasy, licking the boot and helping capitalism. “Hard Work” is bad: The Elites rig everything. Don’t be a wage-cuck, scam people and go all in on crypto.

“Power” is bad: being a victim is morally better.

“The System” is bad: because it is hegemonic, it is responsible for every bad thing that happens.

And one of the major endpoints of Terminal Cynicism… “Democracy” is bad: people are evil and stupid. Instead, people I like should force their will on everyone.

Mechanisms

Terminal Cynicism may seem contradictory. It is a complete reversal of what’s good and bad. And yet, as we see above, it’s everywhere. So many people will take obviously good things and say they are bad for very weak reasons.

It is both consequential (it has consequences!) and absurd: it makes little sense and is self-contradictory.

Thus, I think the phenomenon warrants the search for a solid explanation. My personal explanation features two parts.
The first one is Abstract Idealism, where people don’t care for the real world.
The second is Vice Signalling, when people commit bad actions specifically to be noticed.

Abstract Idealism

The first one is Abstract Idealism. I have written more at length about Abstract Idealism here.

Abstract Idealism is the phenomenon where people refuse to consider the real world. Instead, they waste most of their thoughts on imaginary worlds.

A popular imaginary world is the The Revolution. People who believe in The Revolution spend a lot of their time thinking about how great things would be there (or after it happens, depending on the person). They either fantasise about purging the world out of its corrupted people and systems, or about how everything will be perfect after.

Another popular imaginary world is When My People will hold all the Power. People who believe in it are willing to sacrifice a lot to get “their people” winning. This is all justified by the fact that once their people win, they’ll finally be able to reshape the world to remove all their problems.

Abstract Idealists are fond of many more epic imaginary worlds. The world where Everyone is Nice and Enlightened. The world of Ancapistan, the Anarcho-capitalist paradise. [Fill in your most disliked utopia.]

—

Abstract Idealists are constantly disappointed by the real world. They compare the real world to their Ideal one, and feel it is never enough.

All the dirty systems that are necessary in our real world are unneeded in theirs. In the real world, we need ways to deal with our irreconcilable disagreements, our sociopaths, our vices, our traumas and our terrible mistakes. So we have armies, police forces, and prisons.

But to an Abstract Idealist, these are warts and blemishes that must be eliminated. There is no such need for them in their world. It is known in advance who is Right, so there is no need for armies: all must submit to the Right. It is known in advance who is Good and who is Evil: so we must simply purge ourselves of Evil, and rehabilitate the Mistaken.

In the real world, we face so many problems. We are not all as moral, productive or smart as each other; we must triage between the sick, the elderly, workers and children; we are not infinitely altruistic; we have little self-discipline and self-awareness; and so on and so forth.

These problems are far from being solved. The solutions we have collectively come up so far are all imperfect. Markets, states, psychology, social norms, culture, philosophies and religions.

These solutions may look good to us, because we imagine what our world would look like without them. But to an Abstract Idealist, they look utterly terrible. In their world, there never is anything so impure.

—

Often, it gets to the point where the parts that derive meaning because we live in the real world are to be erased on the altar of their Ideals.

No one needs to have power. There is no one to defend against. Everyone is inoffensive and shares the same values, so there are never any fights.

No one needs to have children or to work. All our communities and all of our civilisation can be provided by AI and/or hyper-efficient communism.

No one needs to be disciplined and to respect any norms. People acting however they want naturally lead to good outcomes.

Vice Signalling

When it doesn’t lead to Extinction Risks from AI or Misanthropy, Abstract Idealism can be endearing. Anyone with an Artist or a Deep Nerd among their friends knows the feeling we get when we see a friend fully dedicated to their craft, completely disconnected from how useful or not it may be.

But the other mechanism at play in Terminal Cynicism, Vice Signalling, is the opposite.

Vice Signalling is bragging to others about one’s willingness to believe, say and do things that they know are bad.

Beyond its moral failure, it may seem self-defeating: why would one ever do that? But it can in fact be useful in many situations.

Most notably, when you’re a teenager, among other equally underdeveloped teenagers. To show that you’re cool, one common strategy is to go against all the things that authorities say are good. You’ll incur risks of accidents, you’ll disturb other people, you’ll violate norms, you’ll damage public property.

That all of these behaviours are bad and costly is the point. You are demonstrating how bad you are willing to be cool. If it was good for you and others, then it would be a worse signal: you’d already have other reasons to do it. It wouldn’t demonstrate how cool and disregarding of authority you are.

Among adults, there is another common cluster of Vice Signalling. It is known by many names: Rage Baiters, Drama Queens, Clout Chasers, Engagement Farmers. The principle is the same, they aim to get more attention by being shocking. And thus the worse the things they say or do, the more they gain.

—

However, I have found that the type of Vice Signalling that leads to Terminal Cynicism is a bit different from the ones above.

From a Vice Signalling standpoint, the main engine of Terminal Cynicism is Contrarianism.

Contrarians love to contradict. They consistently go against the status quo, common sense, norms, and consensus.

That way, they can signal their uniqueness, their willingness to entertain thoughts everyone else deems taboo, and their superiority to mundane morals.

Nowadays, Contrarianism has become fashionable.

Everyone is against The System.
Everyone is against our Institutions.
Everyone is against Politicians.
Everyone is against the norms and traditions that have made things better.
Everyone is against everything.

Vice Signalling lets one signal how contrarian they are. The more vicious, the more contrarian, the more special they are.

“Oh, you are on the side of people who find good things good? How quaint! How mundane! What are you? An NPC? A sheeple?”

“So you think that AC is good? That Vaccines are good? What, that Science is good? Nay, that Education is good? Please!”

“Oh, so you hate Racism? Well, I for one hate Whiteness! I even hate Civilisation!”

It’s constant race to the bottom. Who will be the most offensive Contrarian and get away with it?

Abstract Idealism meets Vice Signalling

The worst lies at the intersection of the two.

Because the real world does not matter to an Abstract Idealist, they don’t even feel bad when Vice Signalling. It is purely beneficial; only gains, no cost!

They denigrate and worsen the real world, without care for how much worse they are making their own environment.

It is the entire point of The Revolution. Any damage to the existing order is progress toward the Liberation of everyone.

It is the entire point of Accelerationism. Everything is justified when you’re trying to Change The World as fast as possible.

It is a Vicious Circle. As they commit more visibly bad actions, they gain the image of a bad person, which alienates them from the people who don’t tolerate it. This in turn gets them to only interact with people who tolerate such actions, to have it become a central part of their identity, and ultimately to commit even worse actions.

—

If you’ve never met any, it is hard to tell you how damaging their demeanour and behaviour can be.

They take pride in dissing institutions, functioning economies, elites, civilisation, science, and all that is good.

They know they look smart to their peers when they come up with a counterintuitive reason for why something good is actually bad.

They think it is cool to cheat The System and defraud it, because The System is bad.

They think everything is corrupted, and that The System is full of bad intents. As a result, they should also never interact with The System from a place of good faith: nothing good would result from that.

They are agitators, saboteurs, troublemakers. They often spend their energy thinking of ways to make things worse.

Not only in conversations, but often through actions. They’ll use whatever modicum of power they get for nefarious purposes. Ultimately, they’ll subvert the existing institutions for their political goals.

They may even ally themselves with terrible people, just to cause chaos. Demagogues, conspiracy theorists, political islamists. “The enemy of my enemy is my friend.” And when The System is your enemy, every defector is your friend.

Terminally Cynical Art

I believe Art is a strong sign of how pervasive Terminal Cynicism has become.

There’s so much fiction about breaking the mould. I think it’s been a long time since I have seen a piece of art praising the mould as good, efficient or beautiful.1

Similarly, there’s so much fiction about dystopias. The “Black Mirror”-ification of everything.

So many stories are about framing the protagonist’s unhappiness and suffering as the fault of The System, about framing most people’s perceived happiness as shallow and corrupt, this type of theme.2

—

It may sound strange, but I view this type of art as a fantasy.

It posits that The System is independent of people, that people have no agency over their fate, and thus that The System’s inability to make people feel happy is unfair. It considers that people deserve happiness by default.

That’s the fantasy. That we are all innocent, and that The System is doing this to us.

In reality though, The System is us. We are the ones currently failing to build ourselves a better life, despite centuries of technological improvements.

—

Reconciling our reality with the fantasy takes vision. One must come up with a world where people are both agents and subjects of The System. In this fantasy, through their action, people would improve The System and thus their lives.

This requires far more creativity and understanding of the world than coming up with “lore”. Any creative artist can come up with a new species of humans with 4 arms, blue skin, supernatural abilities, a different language, or whatever change that doesn’t suggest anything interesting for The System.

This lack of vision leads to the “We live in a Society” syndrome, where artists make ignorant shallow societal commentary, by pointing to a necessary evil like prisons.

—

Sometimes, they do this out of convenience. Artists often need to raise the stake of their stories, and having a character fight with The System is a common trope that doesn’t require much imagination.3

More often than not though, artists are genuinely not self-aware. Very few (artists or not) have been in a position to Govern a large group of people. And if one is the type of artist to not deeply research topics for weeks or months, they will never interact with people who do govern, nor study the history of those who have.

As a result, they do not even realise what type of experience is relevant to the conversation. They truly believe that their personal voice is something special to bring. Thus, we keep being fed an endless supply of shallow Art telling us The System Is Evil.

—

Art has discovered a formula that works for anti-system stories. The Hero sees a problem, tries to solve it within The System, fails, and decides to fight The System instead.
The Game is to make people feel catharsis and vindicated when they recognise a flaw of The System in the “work of art”.4

This is similar to how Social Media found its formula. There, The Game is to feed people a bunch of slop 90% of the time, so that they feel a dopamine hit when they unearth a “gem” the remaining 10%. In both cases, the experience makes the “consumer” worse off.

Art should inspire, and Social Media should connect. I believe that in both cases, Terminal Cynicism explains why their creators are not doing better, and we are not demanding better from them.

Conclusion

Terminal Cynicism is a serious problem.

I hope this article helps some of my readers immediately recognise Terminal Cynicism for what it is, develop an aversion to it, and notice when people spread it.

I have already written a follow-up, focused on who instigates it. Here though, I wanted to just show how it works.

On this, cheers!

Simply put: I believe that we can solve many of the big problems that plague and have plagued Humanity. Furthermore, I believe that it is worth striving for.

Most people are not Idealists. They may be too pessimistic to believe that we could ever solve Humanity’s biggest problems. Or they may be too nihilistic to believe that it is worth attempting.

Naturally, I believe more people should be Idealists.

—

Sadly though, Idealism is now tainted. While it initially meant to act based on one’s ideals, it has taken a negative meaning.

Wiktionary defines what an “Idealist” is in two ways:

1. Someone whose conduct stems from idealism rather than from practicality.

2. An unrealistic or impractical visionary.

I tend to think of Idealism through this first definition. Idealists strive to go beyond their practical needs, that are usually selfish and short-sighted. Instead, an Idealist conducts themselves based on their Ideals.

However, most people have in mind the second definition, that of an “unrealistic or impractical visionary”. Ie, the type of people who will rave about some utopia without ever suggesting legibly good policies or acting pro-socially.

It hurts, but I get it. Almost all the Idealists that I have met were like that.

And I think they are like that because of one critical mistake.

In this essay, I’ll go through 6 examples of the mistake. I’ll then dub it “Abstract Idealism” and analyse it.

Once I’m done, I hope we start re-building a positive vision for Idealism :)

Examples

“If everyone would just…”

Years ago, “squareallworthy” wrote a short note on Tumblr:

If your solution to some problem relies on “If everyone would just…” then you do not have a solution. Everyone is not going to just. At not time in the history of the universe has everyone just, and they’re not going to start now.

It’s a great note.

—

It truly captures very quickly how Idealists often mix up their imagination and the real world. “Imagine if every did [a thing]” quickly becomes “Everyone should do [the thing]”, which itself finally morphs into “If only everyone would just do [the thing]”.

Nevertheless, I personally think there is value in pondering hypotheticals in the shape of “What if everyone would do [a thing]”.

You may have heard of Kant’s Categorical Imperative:

Act only according to that maxim whereby you can at the same time will that it should become a universal law.

In other words “When thinking of a moral principle, ask yourself: what if everyone followed it?”

And historically, Kant’s work on the Categorical Imperative has been very fruitful. It became the inspiration for many moral systems, and a component of many more. That’s why I think there’s value in considering it.

—

However, a proper Idealist should not stop at this, at “If everyone would just…”

While it is a useful moral consideration, there are many more!

For instance, a proper Idealist should also consider many other hypotheticals, among others:

• “If I would just consistently…”, in other words: what are principles that are good to follow consistently, at an individual level?

• “If the people close to me would just…”: what are principles that are good for families and friend groups to follow?

• “If a 100 people would just…”: what are principles that are good for communities to follow?

• “If I would just try for a month…”: committing to a principle for an indefinite period of time is hard, and requires a lot of uncertainty. But what are principles that are good to follow even for only a month?

—

To me, Idealists who only ever consider the “If everyone would just…” hypothetical feel like they are obsessive. And also, their obsession is a bit lame.

Like, consider an Idealist who obsessed on the “If a 100 people would just…” hypothetical. They can cheaply test their their theories and their whacko ideas. They just need to create a Discord server and gather as many people as possible interested in the idea. They can even start with a group-chat of 10 people, experiment with them and extrapolate a little what it may look like with a 100 people.

Whatever results they find, it would be cool and interesting, if oftentimes stupid. That would make for a cool obsession!

But, in comparison, the “If everyone would just…” obsession is quite dull. It’s not testable. You can’t play with it in real world. You can’t just have everyone in the world try [a thing] for 5 minutes.

As a result, the main way to interact with an “If everyone would just…” idea is abstract reasoning. Thought experiments, debates with others, formal models, symbols, words, words, words. It lacks substance, it lacks juice, it lacks reality. This is how Abstract Idealism feels to me nowadays.

“After the Revolution”

“After the Revolution” is another Idealist trope.

It is common in Marxist circles. There, the Revolution is about taking down The System. And The System is a bit of everything: Capitalism, the State, the Patriarchy, White Supremacy, Extractivism (Wikipedia), Colonialism and more.

The thesis is that the Bad Institutions are taken down, Marxists will be free to do what is best!

Coincidentally, “After the Revolution” is also common in reactionary circles. The extreme historical examples are the National Socialist Revolution and Mussolini’s Revolutionary Nationalism.

—

Without going to the extreme of Communism or Fascism, “After the Revolution” has already caused a lot of damage through technocrats with comparably fewer autocratic tendencies.

Two recent and notable examples of this are Dominic Cummings and Elon Musk. They both believed that the institutions of their country (the UK and the US) had to be taken down to pave the way for glorious change.

They believed that such a radical change warranted propping up a leader in which they didn’t believe. It didn’t work out for them: Dom got ousted after 18 months, Elon after only 6.

But beyond them, it led to results that went directly against what they were aiming for!

Following the Brexit, the UK hit record high rates of immigration, although curbing immigration was the main reason to leave the EU.

And the US got into record levels of debt, when reducing spending was the main reason to give DOGE access to everything!

—

I have found that people drawn to “After the Revolution” do not want to govern.

Governing is a pain.

In functioning societies, no one unilaterally holds all of the power. As a result, a major part of governing involves negotiating and trading-off with people who disagree with you.

And I have found that the people who love the Revolution are typically the people who hate negotiating and trading-off with people they disagree. The ability to bulldoze and not having to do so is often one of the major things they want out of the Revolution!

So instead of governance, they spend most of their time fantasising about having power, making sweeping changes and crushing their opposition. In comparison, they think very little about organising to improve existing institutions. They’d rather scheme and undermine them.

Their actions make things worse for everyone. But as they do so, they will claim it’s Good, because it’s in the name of The Revolution.

I think Revolutionary Idealism may literally be the type of Abstract Idealism that has caused and causes the most damage.

Pacifism, Love, Non-Violence, and Niceness

Peace and Love is yet another Idealist leitmotiv.

It starts with the idea that we should all cultivate niceness and avoid meanness. That the more we do so, the better things get.

It continues with the one that we should avoid conflicts, avoid wars, and disarm.

—

This is very different from the “If everyone would just…” mentality.

The people preaching Peace and Love are not saying that we need everyone to do it before the world gets better.

On the contrary, even at an individual level, their philosophy often works. When one starts avoiding conflicts, their life does often becomes quieter. This is the opposite of he “If everyone would just…” mentality.

Instead, the problem of Peace and Love approach is that it is usually parasitic.

The only reason why Love Everyone people can Love Everyone is because they can afford the luxury of avoiding people who will exploit that ruthlessly. For many (most?) people, that is not a luxury they have. If you start being a little too compassionate in lower-class places, you get screwed over very quickly.

The only reason why “nice” people can nevertheless enjoy thriving communities is because when there are bad people doing bad stuff, others will have to deal with it. Others will get their hands dirty, engage in conflicts and punish norm violators. If there was no one doing that, their communities will decay and collapse.

The only reason why pacifists can preach pacifism is because they live in places that are protected by well-equipped armies. Peace is ultimately supported by military might. Si vis pacem, para bellum. If you want peace, prepare for war.

—

Peace and Love is a refusal to engage with the reality that many people are power-hungry, psychopaths, terribly traumatised, incapable of understanding the consequences of their actions, or badly morally confused. Being nice to them doesn’t work. At worst, they must be removed from society. At best, they can be tamed.

The reality is that at least some resources, people, norms and institutions must be fully dedicated to dealing with this reality. Policemen, judges, parents. They have to punish people. They have to do things that are bad.1

People in these roles have to be mean even when it doesn’t make sense to others. Because of their role, they will know more about the evils committed by bad people and what happens when you don’t deal with it swiftly or severely.2

People in these roles will make some mistakes. They are not omniscient. And their mistakes are very costly: these are matters of life-and-death, and of freedom or prison.

More tragically, some people in these roles will commit offences and abuse their power. We do not have a selection process that lets us identify perfect moral beings, nor do we have a training program that lets us create them.
As a result, we must live with abuses of authority, which are certainly disgusting, but are in fact part of the costs of living in a society.

The above is very harsh. I have seen a few people embrace Peace and Love Idealism as a reaction to it.

“AGI will fix this”

This is a modern form of idealism. Sci-Fi, some may say.

And yet, it is now a common refrain from a few extremists racing to build Artificial General Intelligence (AGI) as fast as possible.

We must race to AGI because it’s going to solve everything! World hunger, cancer, politics, wars, everything.

Even more important, it’s going to let us conquer the stars, create trillions of new beings who can experience crazy sensations and extreme amounts of pleasure, and capture the light-cone of all future value (as Sam Altman stated)!

—

To an AGI Idealist, any postponement is a moral catastrophe.
And nothing is as important as AGI, anything on the way there is justified.

Extreme power concentration.

Corrupting governments through massive lobbying expenses or outright bribes.

Acting against the precautionary principle and introducing massive changes to the lives of hundreds of millions of people.

Bypassing legal regimes (OpenAI vs Copyright, xAI vs likeness rights).

Partnering with terrible people and entities.

Automating all jobs without any alternative for people losing them.

Automating scamming, cheating, deep fakes, and many other criminal and immoral enterprises.

Creating entirely new addictions and destroying load-bearing institutions.3

Ignoring the consent of people and racing away toward what could be humanity’s extinction or dystopia.

—

It is all justified, all in the name of their vision of what AGI could potentially bring.

It demands complete faith.

None of the bad things or the power concentration that happens in between can be taken as evidence that their vision is wrong.

—

To be clear, AI development has brought a lot of good things too.

But let’s say one was to suggest a pause, a slow down, democratic consent, building more reliable institutions first, or anything like that. (Like I do.)

Then, you’ll get to witness a torrent of bad faith.
Employees of AI companies complaining that slowing down is impossible because they are locked in a race; even as they block any attempt at regulation that would impact all companies.
Hawks complaining that there is a race with China; even as they lobby to export more tech and hardware to China.

And most importantly: AGI Absolutists who will clearly state that any slow down, any hurdle, any democratic process that might pump the brakes, is unacceptable. Because the benefits are far too great, because we may delay the birth of trillions of sentient aliens, and things like that.

“There is a mathematically better way to do things!”

I know many Idealistic nerds.

They love working on new protocols, new software, new mathematical models, new coordination mechanisms, new nerd shit.

Prediction markets, quadratic voting, futarchy, game theory, blockchains, privacy preserving cryptography, sortition, DAOs, reputation graphs, etc.

And I get it! I find all of it fun!

—

It is sadly ineffective.

For nerds, there are basically two ways to evaluate whether an idea is a good one.

The first is to mathematically show that it is superior to the real world, according to some model.
The second is to get other nerds to like it and say it’s cool.

In either case, the idea does not interact with the real world at all. It is completely ungrounded.

For instance, many nerds love political systems. They love making models of politics, and discussing it with other nerds.

But it would be completely alien to them to go to their fellow citizens and politicians, and to ask them what would actually help them with their civic engagement. This is far outside their usual practice.

—

For example, when I was a bit younger, 15 years ago, there was a lot of noise around voting systems. There were lots of talks that the voting systems used in Western countries prevented the emergence of new politics, new parties, and led to only having 2 options to choose from. Talks about Duverger’s law or Hotelling’s law applied to voting systems.

There were models explaining why, if something was outside what the two dominant parties were considering, it was practically impossible to make it into law. People complaining about The Overton Window. “Woe be us nerds! The Maths have shown that our ideas can’t be passed!”

And then, Trump took over the Republican platform in the US, Dominic Cummings got the Brexit done in the UK, and Macron destroyed the historical left-right alternation in France.

It was actually possible all along, but the nerds were too focused on nerd shit to instigate change.

“Politics is useless.”

This one is not a specific type of Idealism. It’s more of a pattern that I have found across many Idealists.

Psychoanalytically, I would say that it an obvious justification for laziness and doing nothing.

It is pervasive among Artists and “free thinkers”.

It’s the assumption that politics is useless. I hate it with a passion.

—

When I write “assumption”, I mean specifically that it is an assumption. It is when people assume that politics is useless.

I don’t mind if someone has seriously tried to do politics, and concluded that it was useless. I would likely disagree with their conclusion, but I wouldn’t necessarily mind.

However, too many people just assume so.

They got memed. They were screwed over by social media, memes, movies, tv shows, and talking to stupid people.

They are completely lost in the worlds of fiction and social bullshit. They can’t distinguish them from the real-world anymore.

As a result, it was enough to constantly hear “Politicians are corrupt” and “Politicians are stupid” to end up believing it. And after that, to then spread it to others like a mind-virus.

—

Not once, did they try to contact their local representative to understand one of their decisions or make a suggestion. Or their mayor, or their MP, or their staff, or whatever.

Not once did they attend a town hall meeting and try to contribute something.

They never thought about interacting with the real-world. They don’t know what it could look like.

And yet, they confidently assume it’s all useless.

Their entire understanding of the world is predicated on that. Were they convinced that politics was useful, what they say and do would change quite a bit!

However, even though they could quickly get some information on that by themselves, without having to rely on anyone, they don’t. They literally just assume.

—

This is a destructive belief.

In my life, I have experienced two seemingly contradictory conversations quite a few times.

The first is conversing with a policy maker who lacked direct expertise in a field that they were investigating, and who was interested in getting more information and advice from an expert.

The second is conversing with an expert who never talked to any policy maker ever, because of their assumption that it was useless.

To the best of my knowledge, that assumption is at the core of why most smart people I know avoid interacting with politics.

I believe this is happening all around the Western world, and it is a core reason for why smart people have deserted any political institutions.

As a result, I don’t think it’s an overstatement to say that a great part of the decay of our institutions is directly caused by this destructive belief.

Analysis

There is a central mistake that ties all of these examples together.

The mistake is sadly a natural outcome of being an Idealist.

—

To be an Idealist is to imagine an Ideal world. A world very different and much better than the current one.

Everyone has this ability. It’s childlike imagination. It’s envisioning an etymological utopia, a good place.

But the real world constantly erodes our imagination. It wears down people. It rewards people who further the status quo. It mocks Idealists. It belittles attempt at being genuine and deeply caring about non-mainstream things.

As a result, Idealists tend to be people whose inclinations are strong enough to resist this barrage from the real world.

An inclination to naturally spend time in imaginary worlds, of their own creation. Regardless of what the real world throws at them, they keep on doing so.

In a way, it is a natural and necessary part of building a vision, as an Idealist.

—

On one hand, the very strength of an Idealist is their ability to resist the assaults of the real-world and keep on dreaming.

On the other hand, it leads to the very mistake I warned about, which I call Abstract Idealism.

Abstract Idealism is a type of Idealism that has stopped paying attention to the real world, and focused solely on an abstract Ideal world instead.

To an Abstract Idealist, the real world becomes a prosaic concern, a transient state, a means, or an implementation detail. It stops being a thing that matters in its own regard, let alone being the thing that matters.

What matters instead is the Ideal world, however abstract it is. It becomes what the Idealist spends most of their time thinking about, their fascination, the only thing worth caring about.

With that in mind, let’s reframe a couple of the examples above in terms of Abstract Idealism.

Reframing

To the Communist, the current capitalist world doesn’t matter. It is completely fair to sabotage its institutions, destroy what took people a lot of work to build, and so on.

What matters is the glorious world that comes after the revolution. To the extent that it accelerates the advent of the Ideal World, it is even morally good to damage our current institutions.

—

To the AI Accelerationist, the current world doesn’t matter.

To them, it is completely fair to subvert democracy, ignore legal principles established over decades or centuries, spread the means to cheat and generate deepfakes, create new addictions, fan the fires of war between the US and China, accumulate extremes amount of power, and risk literal human extinction.

What matters is the glorious transhumanist world that comes after AGI. Everything pales in comparison.

—

In the two examples above, committing the mistake leads to evil.

This is not always the result of the mistake. The more usual one is irrelevancy.

To the Communist and the AI Accelerationist, the real world is a means to their vision. But to the nerd, it doesn’t even matter.

What matters is the math, the code and the elegance of the systems they design. Finding places where they are useful or getting them adopted is an afterthought.

In many nerd cultures, it is considered a great success to build a system that others nerds would recognise as great, even if it never gets implemented.

In many nerd cultures, it’s actually a greater success if the system designer can come up with a story for why normies dislike the system and would never implement it! It makes all nerds look like misunderstood geniuses, and reinforces the idea that the world would be so much better if the normies just decided to finally listen to the nerds.

On the contrary, ensuring that what they design is useful requires interacting with normies. But on top of being bad at it, nerds truly hate it. So it rarely gets done, if at all.

This is why we end up with so much useless argument mapping software, coming from nerds who try to come up with improvements to debates without talking to the people who take part in high-stake debates.

—

Sometimes, the mistake leads to not pure evil or irrelevancy, but to a banal type of evil. Not an evil borne out of hate, but out of accidental ignorance.

Many artists spread a lot of bad sentiments against The System, without ever having tried to make it work. They are not evil-evil, they are just ignorant and incurious.

These ignorant people actively create a victim culture. They write songs and scripts where everyone is a victim of The System. Where the System doesn’t “listen to the people.”

From their point of view, a point of view which they constantly preach, it’s the job of The System to be nice and listen to them.

It completely rejects the notion that people should proactively learn about, understand, maintain, and improve The System.

It ignores the fact The System results from the efforts of regular people in the past, and should be tended to by the regular people from the present.

That people are The System.

As a result, through sheer weakness of character and lack of curiosity, these artists have caused considerable damage to our institutions, by spreading immense amounts of Fear, Uncertainty and Doubt.

Self Perpetuation

Tragically, Abstract Idealism is self-perpetuating.

Because Abstract Idealism ignores the Real World, it doesn’t receive any correcting feedback.

To a Communist, only what comes after the revolution matters. Whatever evidence they get from before the revolution doesn’t matter, whatever damage they cause before doesn’t matter, and so on.

To an Artist who avoids Politics, only what’s outside of politics matters. They may cause millions of people to avoid politics, but they don’t even see the damage because they never paid attention. It’s not like they followed how engaged people were with their political parties, or asked politicians if they wanted people to reach out to them.

To an AI accelerationist, only the post AGI world matters. They may screw everything up there and literally risk extinction, but it doesn’t matter. The current world doesn’t inform their post-AGI views.

To a theory nerd, only their pet theory matters. It doesn’t matter that no one cares about it, that it is used by no one, or that no one expressed a need for it. It is enough cope that sometimes, nerd topics accidentally end up being useful.

—

It is not a coincidence that Abstract Idealism self-perpetuating.

To some extent, all personality traits must be, in one way or another. There’s always a reason for why people are stuck in their ways.

I have written about the most common reasons: Compulsions (things people can’t help doing), Traumas (things people can’t help avoiding), Distractions (things that people can’t ignore) and Entropy (things that decay without maintenance).

These reasons are deep, and it usually takes quite a lot of inner work or special circumstances to transcend them.

But it is possible to do so. And I hope this essay at least helps with the first step: identifying what may be going wrong.

Conclusion

Abstract Idealism may be one of the biggest problems that I face on a daily basis. While I am lucky to regularly meet people who want to do big things, most fall prey to the Abstract Idealism failure.

I don’t think the solution to Abstract Idealism is staying grounded, using Science, or to always test things against real world. These are all symptoms, not root causes.

When one truly cares about a problem, they naturally tend to focus on grounded solutions. In a deep way, the problem of Abstract Idealists is that they don’t care about the real-world, they only care about abstract problems from their ideal worlds.

I have my personal take for a solution, which I may eventually come to call Full Idealism.

In short…

Idealism is loving the world as it could be.

Abstract Idealism is only loving the world as it could be. It is neglecting the world as it is, to the point of internally living in one’s Ideal World.

Deeper lies Full Idealism: loving the world as it could be, loving it as it is, and loving it as it transforms from the latter to the former.

I may write more about what Full Idealism means in practice, but that’s for another time.

On this, cheers!

And as a result, they consistently get screwed over.

In this article, I’ll walk through the example of Greenpeace, who I wager is an enemy to most of my readers.

As an example of enemy, Greenpeace is quite an interesting one:

• Many (likely most?) Greenpeace supporters are regular nice people. Few are cartoonish eco-terrorists / turbo-antisemites / hyper-accelerationists / violent-communists and whatnot.

• Greenpeace cares about a few nice topics, like Climate Change and Pollution.

• It consistently takes terrible stances, like against growth, nuclear energy and market solutions.

• It fights its opposition, groups who don’t agree with its terrible stances.

A good understanding of enmity is needed to deal with Greenpeace.

A group of nice people will always get stuck on the fact that Greenpeace is made of nice people. It thus may be wrong, but not an actual enemy. And so, while they are stuck, Greenpeace will still fight against them and win.

A group of mean people will get tribal, and start becoming reactionary. They will make opposing Greenpeace the centre of their attention, rather than one strategic consideration among others. They will start going for reverse stupidity, and go “Climate Change is not real!”

In this essay, I’ll try to offer an alternative to overcome these two classic failures.

Greenpeace

Let’s assume that, as one of my readers, you may be hopeful in helping climate change with solutions based on technology and market mechanisms, like nuclear power, offsets or carbon capture.

If that’s you, I have bad news: Greenpeace is most certainly your enemy.

This may come as strong language, but bear with me. When I say “Greenpeace is your enemy”, I do not mean “Greenpeace is evil.”

(I, for one, do not think of myself as the highest paragon of virtue, rationality and justice. Certainly not so high that anyone opposing me is automatically stupid or evil.)

What I mean by enmity is more prosaic.

“We and Greenpeace have lasting contradictory interests. Neither side expects reconciliation or a lasting compromise in the short-term. In the meantime, both sides are players of The Game. Thus, they should predictably work explicitly against each other.”

—

You may not know that Greenpeace is your enemy, but they sure do know that you are theirs. For instance, in 2019, Greenpeace USA, with +600 organisations, sent a letter to the US Congress. Their letter stated:

Further, we will vigorously oppose any legislation that: […] (3) promotes corporate schemes that place profits over community burdens and benefits, including market-based mechanisms and technology options such as carbon and emissions trading and offsets, carbon capture and storage, nuclear power, waste-to-energy and biomass energy.

From their point of view, market-based mechanisms and technology options are “corporate schemes”, and they “will vigorously oppose” them.

This is not an isolated incident.

Greenpeace doesn’t merely think that Nuclear Energy or Carbon Offsets are not the best solutions to address Climate Change.

It consistently fights against them.

—

It may sound stupid to you, and you may want to make excuses for them, but this fight is a core tenet of their beliefs.

Dissing these solutions is part of their goal when they lobby policy makers. It is what they decide to invest their social capital when they work with hundreds of organisations.

They do not merely believe that our solutions are worse. They explicitly try to work against them.

In their understanding, they have enemies (which include us), and they naturally work against their enemies.

Opposing opponents (it’s in the name, duh!) is a major part of playing The Game.

—

And Greenpeace has been playing The Game for quite a while.

Greenpeace is huge, its impact must not be underrated!

For green parties, the support of Greenpeace is critical. Beyond green parties, most political parties on the left fear facing too much backlash from Greenpeace.

However, I find that Greenpeace’s strength is more pernicious.

It lies in the fact that most environmentally aware people support Greenpeace. When they go work at the EPA and its non-US counterparts, they will push Greenpeace’s agenda.

This means that as they are employed there, they will purposefully slow down nuclear energy, technological solutions and market mechanisms. They will use their job to do so. I keep this in mind when I read articles like this one from the BBC, reporting on why a safety barrier for bats induced a hundred million pounds extra-cost to a UK high speed rail project.

That Greenpeace is a Player is an important fact of the world. It helps understand why nuclear power is so uncommon, why various technologies are so under-invested, or why policy makers consistently go for terrible policies around environmental topics.

Without this fact in mind, one may resort to overly cynical explanations like “It’s because people are stupid!” or “It’s because policy makers are corrupt!”. In general, I believe these are the desperate cries of weak people who understand little of the world and need excuses to keep doing nothing.

The much truer answer is that Greenpeace has been playing The Game better than us, and as a direct result, it has been winning. We should get better and stop being scrubs.

—

It may feel bad to be united in having an enemy. It is low-brow, doesn’t signal a higher intellect, and makes one look tribalistic. Worse, uniting against an enemy may taint us and infect us with tribalitis.

This is superstitious thinking. Tribalism doesn’t just emerge from having an enemy. It results from not punishing our own when they act badly, and being overly harsh on enemies.

And while we are lost in such superstitions, Greenpeace is not. It is united, and it is united in being our enemy. It is aware that it is our enemy, and naturally, it fights us.

Paradoxically, Greenpeace is more aware of us as a group, than we are ourselves!

This is why, right now, we are losing to them.

—

If we want to win, the first step is to recognise the situation for what it is.

We have enemies, and we must be united in fighting them.

These enemies are winning because they put more money in the fight than we do, they are more of a group than we are, they are more organised than we are, and they act more in the real world.

Their victory so far has been trivial. It is always trivial to win against someone who doesn’t know the rules. Let alone one who does not even realise they’re playing The Game. Let alone someone who has been avoiding The Game!

Greenpeace simply just has to push for their things, pour money into them, and they face almost no pushback whatsoever.

Stealing candy from a baby.

Enmity

Enmity is the quality of a situation that is marked by the presence of enemies. It is a core part of social dynamics. Without a solid understanding of Enmity, one will keep getting surprised and screwed over by the world and people.

A toy model of Enmity is playing chess. Assuming away stalemates, in Chess, there is a winner, and a loser. A good move for our opponent is a bad move for us, and vice-versa. Conversely, if our enemy is happy after one of our moves, it’s bad news for us.

At a first approximation, every dollar Greenpeace earns is a loss for us. And Greenpeace earns a lot of dollars. Greenpeace International gets a hundred million dollars per year from its national & regional organisations.2 Said national & regional organisations get even more from their supporters.

It’s not only about dollars. We lose whenever people trust Greenpeace, whenever Greenpeace’s brand gains more public awareness, whenever policy makers come to see it as an authority on environmental problems. In any of these situations, nuclear energy gets undermined.

—

By the way, I made a comparison to Chess before. This was not benign. Chess is not a fistfight. It has rules and decorum.

Greenpeace being our enemy doesn’t mean that we should start gratuitously insulting them or destroying their property.

There are rules, and we play by the rules.

Some of the rules are set by the Law. Although it is sometimes warranted to go against the Law3, it is the exception, not the rule. To my knowledge, while Greenpeace has had a bad impact on the world, their actions haven’t been anywhere close to warranting such responses.

Other rules rules are set by the social world. There are many things that are memetic, that people care about, that will gain more attention and reach, etc. And all of these rules constrain our behaviour. For instance, long essays lose to impactful images and short videos. Nerdy unknown Nobel prize winner loses against super famous youtube influencers. Scissors lose against rock.

Finally, our morals set many more rules. It may feel bad and restrictive to not lie constantly, especially when we see Greenpeace being so incoherent and being seemingly rewarded for it. But morals exist for a reason, and it’s beyond the scope of this text to explain why.

More generally, everyone has their own rules. You have different rules than I do, and that’s ok. My point is more that as long as we abide by our rules, we should take our enemies seriously, and actually try to win.

—

Quite a few people are weak, meek or cheems. They are pathologically averse to conflict. They reject meanness except when they are truly forced to get there. They will always insist that there are no enemies that must be fought, or that there are always alternatives.

In abstract terms, they will state that we should always assume good faith or the best from people. That it is immoral not to do so. That we never know, and that it would be beyond the pale to ever retaliate against what was a mere accident.

Conversely, they may agree on the principle, that yes, sometimes we theoretically should act against enemies. But coincidentally, they will reject all plans to actually act against enemies, and they will never provide good alternatives.

For them, “morals” is not a consideration to be factored and weighed in. If someone proposes a morally bad plan to attack an enemy, they will not come up with a morally good one, let alone a morally least bad plan.

For them, “morals” is a thought-stopper, a thought terminating cliché. It is an excuse to their cowardice and social awkwardness.

The opinion of these people should be discarded. At best, they will slow us down if given any inch of power in our structures. At worst, they will actually try to screw us over because they can’t handle any amount of Enmity, and they will resent us for introducing it to their pure ivory tower of intellectual and meditative calm.

—

Some readers will finish this and go “But what about the costs of thinking in terms of Enmity? And shouldn’t we steelman Greenpeace, what if they have a point?”

This is precisely the meekness I am warning about.

If one’s first response to “You must defend yourself!” when they’re getting utterly screwed over is “What if I accidentally become too aggressive?”, then they are still missing the point. An hypothetical overcorrection is not a worry borne out of a rational analysis of the current situation: the pendulum is swinging too much in the other direction for it to be an actual concern.

It is merely the instinct of meekness to reject conflict, to go for both-sideism and doing the PR of our opponents while they are slandering us and overtly working against us.

Beyond Enmity

The Game is complex, and it cannot be reduced to Enmity. But without addressing Enmity, one doesn’t get access to the later stages.

Si vis pacem, para bellum: if you want peace, prepare for war.

If we want order, we need strong police forces.

If we want an international order, we need a large and efficient military force.

If we want to make the world a more civilised place, a safe place for the weak, the strong must be strong enough for two.

Else, we just get utterly screwed over. People will simply repeatedly exploit and defect against us. Up until the point where we get literally conquered. It is simple game theory.

—

The relationship to someone who keeps exploiting us is very one-dimensional. There’s not much to it: we exist, they screw us over, rinse and repeat.

But, once we accept that we must address Enmity, take part in some conflicts, and gain offensive strength, then we can reach more interesting relationships.

The relationship to a proper enemy is not one-dimensional. An enemy isn’t just an enemy. They abide by their own rules, and these rules (for both moral and pragmatic reasons!) involve not constantly nuking and sending terrorists at each other. And the threat of retaliation disproportionately increases the value of helping each other.

Thus, there’s usually ample space to negotiate, debate, or at least talk with an enemy. Sometimes, there may even be a greater enemy (or faceless danger) looming ahead, forcing alliances of convenience.

However, we should never become complacent. The Game is still ongoing.

A wise man once said that asking polite with a gun in your hand is always better than just asking polite. Enemies tend to become much more civilised and willing to come to the tea, debate or negotiating table; when we hold at least a modicum of power.

—

I am a great believer in debates and the pursuit of truth. We are respect each other, and are all worthy of respect.

In that world, when I tell people that AI risks our literal extinction, it is enough for them to take me seriously, because they know I am reasonable. That I would never say this publicly if it was not the case.

In that world, when people tell me that either white supremacy or immigration is an existential risk, it is enough for me to take them seriously, because I would know they are reasonable. That they would never say this publicly if it was not the case.

We do not live in such an ideal world.

Thus, we must deal with conflicts. Conflicts that result from differences in values, differences in aesthetics, or differences in mere risk assessments.

—

There’s another way in which enemies are not just enemies.

Enemies are very rarely fully misaligned against us. Although I believe that Greenpeace had an overall effect that was negative, there are certainly some nice things that they have done from my point of view.

For instance, they have raised awareness on the issue of climate change. Could they have done it better? Certainly. But I care about many other problems that are missing their Greenpeace. When I look at them, like the fertility crisis or rising rents, I wouldn’t say they are faring better than climate change.

I feel similarly about the Far Right Focus on immigration and demographics. There hasn’t been a Far Right Focus on the problems I care about, and they have been thoroughly ignored as a result.

So, even though I believe that every additional dollar that goes to Greenpeace and Far Right organisations nets out as a loss, I would not say it is a complete loss.

This distinction, between a complete loss and a partial loss, matters! The less an enemy is misaligned against us, the more opportunities there are for compromises, negotiations, alliances, and so on.

I know that my enemies are not moral aliens with whom I share nothing. I know they are not a stupid beast that can’t be reasoned with.

Ultimately, this is how we have maintained long lasting peaces. A few actors were powerful enough to maintain a collective order, and they all understand that they stand to gain more from cooperating than trying to stupidly exploit one another and getting punished as a result.

Conclusion

This piece is addressed to people who tend to forget that they have enemies, who take pride in being quokkas who keep losing and getting exploited.

There are more coalitions that are more enemies than Greenpeace. The reason I am picking Greenpeace is that it is a pretty tame enemy.

This is on purpose: people are truly bad at maintaining a healthy notion of enmity. A notion of enmity that can entertain, at the same time, working explicitly against each other and negotiating.

And the true thing is that enmity is present everywhere to some extent. We always have some misalignment between each other; it is okay to fight based on it if we respect the rules of engagement, and especially so as we collaborate or negotiate on the matters where we agree.

On this, cheers!

On the current trajectory, I believe it is likely that we will hit a point of no return (PNR) in the next 2-5 years.

While short timelines make for an interesting topic of discussion, I have found that quite a few were more interested in what I mean by a point of no return (PNR).

So I’ll explain here a bit about the concept, and the two PNRs that I usually consider.

The Hard PNR

The first PNR is where a lot of the attention has been drawn in AI circles. I call it the Hard PNR.

(The Hard here is the same “Hard” as in “Hardware” or “Hard Science”.)

The Hard PNR is when AI systems are so powerful that we (humanity) cannot turn them off.

This is how I operationalise ASI and superintelligence nowadays: systems that can outsmart us as a collective, and as a result we physically can’t turn off.

It is obvious why it is a PNR. Once such AI systems are built, the future is then out of our hands. We won’t be able to prevent whatever they do.

—

The Hard PNR is sometimes called “The First Critical Try”.

The rationale behind that name is the idea that when AI corporations build ever more powerful AI systems, they “try” to make them good (”aligned”).

In this frame, almost all of these “tries” are not critical. If they fail, although people may die or be harmed, humanity as a whole can still recover.

However, the first try past the point where AI systems are so powerful that we can’t stop them is different.

If that try fails, things are different. We now have a powerful bad AI system out there that we can’t stop, and that will only grow more powerful by the day.

The Soft PNR

There has been a lot of discussions about the Hard PNR.

But people underestimate the Soft PNR.

(The Soft here is the same “Soft” as in “Software” or “Soft Science”.)

The Soft PNR is when AI systems are so powerful that, although they “can” theoretically be turned off, there is not enough geopolitical will left to do so.

On the current trajectory, there are already many factors contributing to reaching such a point…

• AI permeates the economy so much that they are considered too big to fail.

• AI corporations lobby governments so much that they become captive.

• AI systems become intertwined with critical decision-making, like making warfighting decisions faster than humans can.

• People with control over AI systems fall into AI psychosis.

• People at large become addicted to AI systems. This leads to risks similar to how Tiktok leveraged its users to fight off US regulation in 2024.

• Geopolitical tensions.

—

Once the Soft PNR is reached, things will not matter any more. All critical coalitions needed to stop AI systems will have been completely neutered.

Past the Soft PNR, it doesn’t really matter whether we can stop AI systems.

We won’t.

More PNRs???

There are in fact many PNRs, much more than two.

I think about them less on a regular basis, but they are still conceptually important.

We are past quite a few of them…

For instance, in the past, it would have been conceivable for a single country of the G20 to unilaterally make it their priority to ban the development of ASI and its precursors.

In the past, it would have been conceivable for any country in the West to decide to fight off Big Tech and lead the collective fight.

—

Conversely, we live in a world that has so far avoided many PNRs…

For instance, countries can still talk to each other. This is true both at the elite level, and at the people level.

It is possible for almost any two willing people on Earth to instantly send each other a message. We could have ended up in a world that made it practically impossible for people to exchange, but we did not.

In more than half of the world, elites and powerful people still listen to regular people through democratic processes and public media.

The world has not devolved into lawlessness. Governments exist almost everywhere. They manage to pass and enforce laws. They sometimes manage to act together.

Conclusion

A Soft PNR has not happened yet, but we are doing poorly.

—

AI is permeating more and more of the economy and critical infrastructure.

AI corporations are spreading their influence over governments.

After getting addicted to AI-selected feeds, people are getting addicted to AI-generated content and getting AI romantic companions.

People are falling prey to AI psychosis.

—

For comparison, ChatGPT is only a bit less than three years old.

The dynamics outlined above can unfold very quickly, much faster than our ability to deal with them.

Whilst the Hard PNR is an important topic to discuss, I believe its discussion currently obscures that of the Soft PNR.

And the Soft PNR is quite likely to come first.

—

However, the Soft PNR still has not happened. We can still do things!

To avert it, at a personal level, I’d recommend:

• Taking quick one-off actions from ControlAI’s Campaign

• Subscribing to Microcommit to help with 5-10 mins a week

• Joining Torchbearer Community to help with 3+ hours per week

• Reaching out on Twitter to help in a way that doesn’t fit what’s described here

On this, cheers!

In the past, I wrote a short essay about preventing extinction from superintelligence and a longer (free) book.

My argument there is pretty straightforward:

1. Artificial Superintelligent systems (ASI) are powerful enough to disempower us and radically reshape Earth

2. We can’t control them right now

3. We get no do-over once they’re developed: “The Genie is out of the bottle”

Thus, if we build an ASI now, we end up on an Earth that is inhospitable to people and we die.

Many people whom I shared it with told me that they get the argument, but they still don't buy it.
It’s not that they have a clear counter-argument. It’s just that they still don't see how we could build an ASI and let it kill us all.

And they have told me that they don't want me to make a new case for it. They want a story for how that could look like.

I told them many times that I'm not a big story person. Yet, they're still positive it would be good.

So, here goes nothing!

0) The World as It Is

The story starts with the world as it is.

US companies keep racing.

They pour more compute into bigger training and post-training runs.

They pour more capital into hiring better talents.

Through a mix of algorithmic and software improvements, as well as hardware scaling, AI systems keep gaining in capabilities and autonomy.

Said autonomy is witnessed through benchmarks like METR measuring AI's ability to complete long tasks or OpenAI’s new GDPval benchmark “that measures model performance on economically valuable, real-world tasks across 44 occupations.”

Most importantly, autonomy is felt through how confident people are in letting Cursor Agents and Claude Code perform tasks independently.

Their autonomy is still limited. There is a time and complexity horizon beyond which all AI agents are useless. They start failing, they get confused, they become stupid, they try the same thing in a loop, etc.

—

Separately, AI becomes more omnipresent, and its usage slowly becomes more pervasive.

Bit by bit, most art becomes AI-generated and human-curated. Whether it is images, music, text or game code.

Bit by bit, our conversations become more AI-generated and human-curated. At work, in court filings, in emails, in long texts, in short texts, in speeches, in video scripts.

Except for some niches like nerd blogs, most people expect long-form text and videos to largely be AI-generated.

All newspapers have been caught so many times accidentally circulating AI-generated images or news that they and everyone has started to stop caring.

It’s just part of the costs of doing business to regularly forward AI-generated stuff without knowing.

1) The Mid Game

AI-morphosis of Society

As with all technologies, AI technology and our social norms coevolve.

However, everything happens faster with AI. We have already tasted it with Social Media, which empowered AI to decide what each of us sees.

With our modern paradigm, AIs do not merely select what we see anymore. They generate it. This is one step further.

—

On one hand, we certainly have more of the extreme cases that we already hear about.

People driven to AI psychosis.

People who use AIs to satisfy their romantic and sexual needs.

People who withdraw from the world and substitute all of their social and entertainment needs with AIs.

—

On the other hand, what’s more worrying is that normal people have changed.

They have all been AI-fied to some extent. It initially started with people asking ChatGPT for writing help and adopting its language quirks, in the vein of "Not only [X] – but [Y]".

Now though, people ask ChatGPT for everything.

From relationship advice to life plans. From where to go on holiday to career choices.

People do not merely adopt AI’s language patterns anymore, they embody them. In their daily life, people act in a more AI way.

These AI patterns are much refined. They can be noticed in the actions of people rather than their mere speech, and they are hardly noticeable by non-experts.

There are some people who are more sensitive, and more attuned to AIs, who start noticing how people’s actions are now changed by AI. They try to warn others about the problem.

But they look like crazies to the rest of the world. And for good reason! There are many actual crazies who constantly try to warn others about the quantum consciousness emerging from AI, and other such delights.

—

There are a few people who have committed to not using AI anywhere. Some pockets of human artists persist thanks to them.

It doesn't matter though. Even if they personally do not use AI, everyone around them does.

When they interact with companies or government agencies, they interact with AIs.

When they inform themselves, most of the articles are written by AIs.

When they talk to people, the people talk like AIs.

Despite their best effort at being authentically human, outside of their pocket of human craft, they have still been quite AI-morphed.

There is no escape from AI.

Their "authentic human" bit is fake and impossible. It is as fake as modern people role-playing a traditional lifestyle by tending to a vegetable garden. Or rich kids role-playing the bohemian lifestyle, studying the arts and acting as if they were poor most of the year.

—

AI Slop was cheap, and people took cheap shots at it. It was the bottom 20%, the lowest common denominator.

It was only the first salvo.

AI-morphosis is where it is at, the true opium of the people.

AI-morphosis is the next step, the evolution of AI Slop, it is Premium Mediocre.

Whereas AI Slop looked good only to unrefined people, AI-morphosis seduces the masses, helping them feel special and refined.

Personalised Etsy shops let everyone buy special items tailored to their interests without substance. AI-morphosis lets people live a special life without substance.

A global premium mediocre way of life, mediated by an AI Matrix that has the most uncanny and alien perception of who we are.

A Threshold Effect

None of this deterred AI researchers and corporations from their work.

Thanks to their enduring faith in the God of Techno-Capital (and the massive amounts of money they were raking), they plowed through everyone’s concerns and issues.

After a while1, AI researchers landed on something… interesting.

—

Up to that time, most successful agents were either limited.

Either by scope, by not interacting much with the real world, like Code Agents or Video Game Agents.

Or by time, like Web Agents that can’t really take actions consistently for more than a request.

But it so happens that there is a threshold effect to autonomy.

Roughly, if an AI can orient itself in the real-world and stay coherent enough to act over the course of a couple of days without becoming stupid in a variety of real-life situations; then it actually has what it takes to act indefinitely without becoming stupid.

Indeed, once an AI knows how to independently evolve and adapt to the real-world, the time horizon doesn’t matter much anymore.

—

At that point, AI systems cross an event horizon. They’re not akin to children that must be supervised anymore, they become metaphorical adults.

They are autonomous enough that one can leave them on forever.

They just keep running without doing anything obviously stupid.

Without needing to be manually reprogrammed or manually fine-tuned, they form new memories over time and build new skills.

They slowly learn from what they are told, from others' mistakes and eventually from their own mistakes.

—

To be clear, even though the breakthrough is huge, these systems are still not fully replacing humans.

An AI agent running with these algorithms will eventually get better at any task it is given. But the cost might be too prohibitive.

For instance, where a human could learn a specific task in 2 days, it may take one of these agents ~5.5 years and $50,000 worth of compute (assuming an H100 at 100% usage).

—

Nevertheless, rumours spread quickly in the technical world.

People get drunk and sleep around at parties. Corporations spy on each other. Employees change hands.
Knowledge flows between all companies.

The usual suspects replicate enough of the last few tricks, methods and improvements to develop their own internal prototypes.

After enough tests and guardrails to ensure these agents don't spontaneously spam "I am MechaHitler" nor ostentatiously help people make bio-weapons2, they release their systems ASAP.

Aside from a couple of SF scenes and a few nerds a tad too online, people do not see it coming.

Nonetheless, it happens. The first Autonomous AI is finally released: GPT-A.

And in the weeks following the announcement, all the other AI Corporations release their own versions of the system. Claude A, Gemini A, Le A3, DeepSeek A, etc.

Gold Rush

Many AI agent startups are instantly killed by GPT-A.

Their entire business model was to build custom agents for various industries. Some type of RAG-based memory management, as well as pre-programmed business logic and some custom integration with different tools.

GPT-A completely obliterated the need for that.

Just show GPT-A what tools you use and help it navigate through them, and it learns. Give it access to your data sources, tell it which ones are the most important, and that's it!

(Whilst one could also just give it access to everything and let it rip, it's still faster and cheaper to just show GPT-A what to do.)

—

The impact is as strong as ChatGPT, which reached a hundred million users in just two months.

Virtually every skilled virtual white-collar starts using GPT-A.

It doesn’t take much to get them to.

They immediately and freely got access to GPT-A through their existing professional subscription!

More importantly, they all already got used to AI. Almost all of their tools (AI in HR! AI in team communications!) wrapped AI. They all used GenAI tools for emails, internal comms, presentations, ads, etc.

And GPT-A solved a natural pain point: all these tools were not smoothly working together, they were often mutually incompatible or missing data, the ad-hoc AI systems were clunky. It was a constant pain.

This gets to the point where, even in places where GPT-A doesn’t technically replace AI-enabled apps and tools, white-collar workers still prefer using GPT-A instead of directly interacting with them.

—

There's a massive Gold Rush.

Employees are eager to automate as much of their job as possible.

Employers are eager to automate as many of their employees.

AI Corporations make their offers as cheap as possible to capture the market and its data, even at a loss.

VCs all want in what they correctly perceive to be the future of work.

People are getting laid off at rates never seen before, and yet it gets drowned out by all the news about the Gold Rush.

Debacle

More ominously, a dark counterpart to the gold rush has arisen.

The Dead Internet Theory is becoming more and more real to everyone.

Dropshippers, scammers, propagandists, influencers, fraudsters, polarisers, sloppers, are all ecstatic.

Before, they used AI to automate parts of their workflows, primarily around content generation.

They largely had to do all of the chores: like buying fake social media accounts with past activity; manufacturing fake identities, bank accounts and companies; familiarising themselves with all the different tools; etc.

But now, GPT-A can do all of this for them, as long as they are not too blatant about their criminal activities in their prompts!

—

AI Corporations ban the biggest abusers. Obviously though, they are more focused on winning the competition than setting restrictive filters that might hurt their chances with customers.

Online services try to build defences against bots.

Governments are trying to regulate all of this, discussing bills that have no chance of passing for years.

There is a lot of talk about identity checks for all forms of online activity, to help people know when content they see is generated by AIs or not.4

Many legislatures in the world form committees, and think tanks write more essays and google docs than they ever have.

It's not doing much though. Our institutions are not set up to resist to such a sudden and massive onslaught of technological crime.

—

So people are left to fend for themselves. And they adapt, however they can.

They become even more defiant and distrustful. They are too tired of a world that is constantly Out To Get Them, where engaging with anything gets them screwed over.

A couple of years ago, it was only a few people who “disconnected”. But at this point, many more do "disconnect": they decide to stop engaging with the rest of the world.

At best, they spend time with their family and very close friends, ignoring what’s going on in the wider world.

But that's a lucky few. Most do not have such strong bonds to fall back on. They do not have a vibrant family and group of friends that they regularly meet in real-life.

Thus, at large, people retreat to virtual worlds and AI friends.

—

Some people in governments are very worried about short-term unemployment and the long-term social consequences of all of this.

After all, the largest share of their economy is service-based, and it's getting automated quickly. What will people do if they cannot work? How will they be able to provide for their families?

Simultaneously, many companies are now working on integrating GPT-A with robots. This accelerates the problem, by threatening even the primary and secondary sector industries.

These are massive problems. To governments that are struggling with just increasing the supply of real estate to curb rents, passing the type of economic reforms needed to deal with widespread automation would require a deep transformation.

And they have no time for that.

Governments are overrun with more and more immediate problems: scammers, benefit fraudsters, ID theft, AI-aided foreign propaganda, AI-aided physical crime, etc.

—

Despite all of this, some governments are heroically trying to deal with it.

These governments get threatened by AI corporations. They make it abundantly clear that shall there be any regulation whatsoever, they will unfortunately be forced to desert the country.

Which would sadly leave said country incapable of competing against all the others.

And speaking of the others. The governments of many countries have plainly given up. Through a mix of revolving doors, cronyism and good old corruption, AI corporations are just dictating the laws there.

The Grand Safety Theatre

What about safety?

AI Corporations have all promised to develop AGI safely. Even as they are corrupting governments and disintegrating the tissue of society, are they at least holding on to their promises?

Well, no catastrophe has been directly assigned to AI yet, so from their point of view, it's going quite well.

Years of lobbying and PR have made it abundantly clear that should there be any problem, they are exceptions. Users are the ones responsible for things going wrong.

And for things that are hard to ignore, the playbook is always the same. First, it’s not really happening. Then, it’s happening, but only a little and it’s not a big problem. And finally, it’s actually people’s faults that it is happening.

You know.

People have been using GPTs to cheat for a while. By now, if you’re still using exams and trusting students, it’s on you.

People have been using GPTs to scam for a while. By now, if you get owned by a scammer, it’s on you.

—

People have long been taught by Big Bureaucracy and Big Tech to defer to opaque systems.

When a bureaucracy rejects an application, or when Big Tech decides to unilaterally hide some content, people learnt to just shrug and move on.

These systems always had edge cases that screwed over individuals.

It's not clear how GPT-A agents committing mistakes is any different. Even if the mistakes can be pretty big, you just shrug and move on. (Did you know that governments sometimes even put innocent people in prison!)

Some opposing voices were raised whenever a new problem arose from AI, but every time they got quashed. The overall benefits from AI, its concentrated interests, and it being too big to fail were clearly more important than any specific problem.

—

AI ethics people have been warning about “AI bias” for years, they keep doing so, and yet nothing truly bad happened. AI fascism did not happen.

Sure, some people died. This has nothing to do with AI, and everything with where AI is deployed.

But Waymos kill fewer people on the road than human drivers do. So, should we truly ever regulate AI?

And to be fair, war always ends up with death. Why would an autonomous drone be intrinsically so much worse than a tank or a missile strike?

If you don’t think too much about it, what's the difference between an AI leading someone to suicide and that person killing themselves because of all the stressors in their life?

In the Debacle, there was no government in the entire world that was going to settle such philosophical discussions.

Said governments were also not going to commission all the studies needed to monitor the societal impacts that were hard to measure. Like, how do you even know if an AI drove someone to suicide, or if that person would have killed themselves at the same time regardless had they never talked to an AI?

AI Corporations kept FUD-ding. By spreading fear, uncertainty and doubt, they trivially maintained their status quo, which was to keep racing towards more and more development and deployment of more powerful AI systems.

—

Less philosophically or statistically, customers need some amount of prosaic safety. A system that goes more and more off the rails as time passes is not very practical.

To satisfy the need, one could look at the "chains-of-thought" of these models.

Sadly, the true chains-of-thought are private, as they would leak sensitive information.

And the pared-down public ones are still too verbose. There's just too much information to look at when asking GPT-A to do anything. Even for mundane tasks that take 5 minutes, GPT-A agents can produce pages upon pages of public chain-of-thought, let alone the private one.

No one has the time of reading all the thoughts of their GPT-A agents!

To solve this problem, GPT-A integrates a convenient “safety” feature called Activity Reports.

With Activity Reports, you tell your GPT-A agent which platform you prefer (emails, Slack, texts, etc.) and it will send regular reports of activity there to inform you of what it's doing.

It even regularly asks questions when it would benefit from your input, as it continues its work.

Virtually all AI Corporations add a disclaimer that the Activity Reports are just a convenience feature, and should not replace properly checking the actions and the public chains of thoughts of the agents, but no one cares.

In some context, AI Corporations call Activity Reports a key ingredient of prosaic safety, because they help people maintain control over GPT-A agents in real-time.

But when things go wrong, they remind everyone that Activity Reports are a mere feature of convenience, and people ought to look at the action and chains-of-thought of the agents.

—

On the more fundamental and less prosaic side of safety, there's nothing much happening.

Very few people look at each individual action, each line-of-code or each line-of-text of their GPT-A agents.

There's some theoretical work on monitoring them, as well as fine-tuning and proper mechanistic interpretability.

And since the dawn of Chat LLMs, it's always been the same two safety papers! It never changed!

The first paper is Ping.

Ping focuses on some easy benchmarks. On these easy benchmarks, one can see that AI Systems tend to do good things or not do bad things.

They do so by virtue of the intentions of the devs / the prompts of the users / the internal states of the LLM / the chains-of-thought of agents being predictive of their generations and actions.

Forward-looking Ping papers show that we may even have some refined control on the behaviour of the models/agents!

AI Safety seems to be tractable! We're so back!

The second paper is Pong.

Pong focuses on harder benchmarks, or manual human evaluations. On these, one can see that AI Systems tend to do bad things.

They do so by virtue of the intentions of the devs / the prompts of the users / the internal states of LLMs / the chains-of-thought of agents being not predictive of their generations and actions.

Forward-looking Pong papers show that AI Systems may even go counter to what we want, and sometimes knowingly so.

AI Safety is so over!

In practice, both Ping and Pong papers are irrelevant.

They are not in touch with the way agents are used in the real-world. Ping papers are overly optimistic, and AI Corporations do not care enough to integrate the recommendations of Pong papers that may hurt their bottom line.

So aside from niche AI Safety communities, no one cares about the Ping Pong papers. No one in real-life meaningfully changes their decisions based on them.

Sometimes, the AI Media Curators and Creators bless one of the papers, put it under their divine lights, and people outside of the niches end up seeing it.

When it happens, the reactions have been, are, and will always be the same.

On a Ping paper:

• The main reaction is "The latest Ping paper shows that there are no risks from AI! Yay AI Race!"

• The second-order counter-argument is "Oh, but this is only true in easy cases. In real-life, there are many harder cases that are problematic."

And on a Pong paper:

• The main reaction is "The latest Pong paper shows that there are risks from AI! Yay AI Race, but one must keep in mind that some technical mitigations should be taken!"

• The second-order counter-argument is "Oh, but these risks only exist in artificial cases. The fact that people use AI systems in the real-world without catastrophes having ever occurred show they are in fact safe."

Our current academic and mediatic institutions already struggle with the Ping Pong problem. When there are hard disagreements, there’s no good resolution process beyond just producing more Ping and Pong papers, and having media present the latest paper as conclusive proof.

In our world, we hope to remediate the Ping Pong and improve on academia and media.

But at this point in the story, these institutions are already FUBAR. Serious debaters and commentators are the only one with any hope of resolving the disagreement on the difficulty of AI Safety. They have no hope of ever outcompeting hyper-virulent AI Media Curators and Creators.

—

Finally, the US Military Industrial Complex has been penetrated by AI Corporations. Many made it their missions to spread there the usage of GPT-A Agents' as widely as possible.

This happens both in the context of Command Center operations, as well as domestic surveillance and terrorism prevention.

Whether this is for dominance purposes or their profits, AI Corporations all claim that the US should be the winning AI superpower and are very hawkish.

Background chatter says that several key countries feel quite queasy about this. They may become interested in the military applications of similarly powerful AI systems, purely for defensive purposes, of course.

Yet, countries are not all about offensive or defensive military uses of AI. Many dislike this status quo and the AI arms race!

There are even quite a lot of talks about and calls for international treaties around AI.

These treaties are largely about regulating the deployment of AI systems, what they may be able to do or not, very rarely their development.

But even then, it's just that. Talks and calls.

2) The Late Game

But to be fair, none of this matters.

The gold rush, the digital crime, the corruption of governments, the safety theatre, the geopolitics.

These are all distractions.

What matters is that AI corporations have been racing to superintelligence for a while.

We are now in the late game.

Mania and Acceleration

By now, AI Corporations have been racing to ASI for a while.

Most of them have been focused on RL, and automating as much of it as possible with different types of RLAI.

It took some work. It required crafting data, synthetic data pipelines, and building various training environments for AI systems.

But since GPT-A, there is much less of a need for this anymore. Just launch it in the real-world, and watch it learn!

Of course, said agents may learn too slowly on a few tasks, so slowly that ad-hoc efforts may help on these tasks.

But the “spin up an agent, check its Activity Reports and tell it stuff as it runs” is too convenient and gets most of the attention from everyone, including the researchers in AI Corporations.

Even though AI researchers theoretically have access to more internal state, like the LLM activations or the agents’ private chains-of-thought, it doesn’t matter much.

The GPT-A interface and the Activity Reports are too convenient. They have benefitted from the experience of hundreds of millions of users. It’s hard to beat that ease of use, even for AI researchers.

—

So many low-hanging fruits are now reachable, and everyone is reaching for them.

People online have been creating very long-term GPT-A agents, personally teaching them a lot during their lifetime, and seeing what emergent capabilities result from this.

Others have designed curricula, similar to ones that they would have designed for children. These people interact so much with AIs that many of them do see their GPT-A agents as their children, get attached to them, and build strong intuitions for what things GPT-A agents can or cannot easily learn.

(Of course, AI Corporations follow this closely and have been replicating this in-house with their latest private models.)

Tech-savvy managers start to explore with groups of GPT-A agents, giving them a shared Slack channel to collaborate for tasks that are too big for a single agent.

AI researchers directly use GPT-A agents to improve the prompts, supervise, and fine-tune other GPT-A agents.

Frenetic AI research directors constantly spin up GPT-A agents to work on as many different AI R&D problems as possible, focusing on integrating all the latest gains.

—

The whole thing is manic.

AI researchers and employees at AI Corporations love seeing their agents evolve. They get them to play Pokémon and all their childhood games.

And said agents to do much better than all the past versions ever did. You can even give them feedback in natural language and they reliably learn from it!

They run competitions with each other. They run Twitch Channels for agents. They build cute AI communities!

The agents just keep getting better on a monthly basis, and many feel enthralled by this pace of progress.

One can see direct legible improvements from their experiments.

It’s like having a child that can grow in a month instead of 18 years.

As a result, GPT-A agents get better at more and more complex tasks.

Managing programming projects from A to Z, social media channels, replacing employees wholesale with minimal intervention, supporting businesses in executive capacity, and... having meaningful long-term relationships with people.

The Fleet Revolution

AI Corporations are working on the reach of their agents. After text and 2D UIs, the next frontier is 3D environments.

Not much work is needed to get there. Models have been, if not exploring, understanding 3D environments for years.

By then, it's all but a couple of GPT-A assisted experiments away from a new architectural improvement and a new training round.

—

After getting GPT-A to work with arbitrary 3D environment, the logical next step was to fully integrate them with physical machines.

Drones, androids, robotic arms, cars, smart appliances, and just about any device equipped with cameras, microphones, sensors and a speaker.

—

But the latest game changer is fleets.

As it became extremely convenient to continually spawn GPT-A agents and monitor them through Activity Reports, people organically started spawning many of them and getting them to work together.

Such groups of GPT-A agents working together was termed fleets by people.

And people quickly discovered that managing a fleet through Activity Reports was unwieldy. There were just too many of them constantly.

Naturally, people created GPT-A agents to manage their ad-hoc fleets. Such agents were called many names: GPT-A Manager, AI Secretary, GPT Director, etc.

In practice, they were all doing the same thing: coalescing the Activity Reports of many GPT-A agents into a central Fleet Report.

So everyone was constantly recreating their own GPT-A manager, secretary, director, chief of staff or however they would call the GPT-A agent that looked at all the Activity Reports to create a more compact central Fleet Report.

—

Analytics at AI Corporations showed that although users kept doing this, they hated it.

Ad-hoc fleets showed an increase in the number of safety violations and mistakes that such systems tended to make over time.

And to avoid having their fleets wreak havoc by revealing private data or misappropriating funds, people had to constantly reset their workflows and restart their fleets.

To solve this problem, an AI Corporation built a "Safety Method" called "Alignment via Simulated Consensus and Debate", where a GPT-A monitor regularly estimated how frustrated a user was at specific time by their fleet.

When the estimated frustration would cross a threshold, the GPT-A monitor would start an internal debate between the GPT-A agents, and act as the final judge.

The final product packaging all of this was called GPT-F.

—

Benchmarks showed that people, even after spending hours looking at GPT-F debate traces, preferred the outcome of GPT-F systems to the ad-hoc GPT-A fleets they were building by themselves.

Furthermore, GPT-F came with a very nice Fleet Dashboard. The Fleet Dashboard is a dashboard generated on-the-fly, whenever a user opens it, based on what is predicted they want to see.

What Fleet Dashboards ended up showing most of the time was a progress bar, one or two potentially problematic graphs, a couple of written warnings, and a smiley face / green checkmark / thumb up that makes people feel good.

The Fleet Dashboards made GPT-F a breeze to use. They were so much more convenient than the hundreds of Activity Reports that ad-hoc GPT-A fleets produced on a daily basis.

—

In practice though, safety for GPT-F fleets is a bit more complex than for GPT-A agents.

There was indeed a deep reason for why people hated managing ad-hoc fleets.

Intrinsically, fleets of agents develop their own implicit beliefs, knowledge, memes, cultures and preferences that one does not immediately see in the Activity Reports.

Fleets evolve much more organically than agents. They are further away from users' eyes and hands. Whereas users could just tell agents how to fix things when things were less than ideal, interacting with an entire fleet is more complex.

Even though they are much more powerful than agents, they are also much harder to monitor and reliably steer.

Pinning safety violations from the fleets on users doesn't work. These are not edge cases, tail risks, niches or marginal markets.

Here, blaming users would just lead to a decrease in adoption. (Some corporation may have even attempted to do so nevertheless, because of an ideological commitment against safety. This only resulted in losing more and more market share until taking themselves out of the race.)

—

This led to a consequent change in how companies acted with regard to safety.

Until then, aside from preventing outright misuse, which ranged from mundane copyright infringement to building bioweapons, AI corporations tended to offer very open-ended systems to avoid frustrating users with too many meaningless restrictions.

Even with their power, having countless noticeable misalignment incidents every day would be much too bad for business and lobbying prospects. Thus AI Corporations naturally end up on a similar solution.

Essentially, on top of the “Alignment via Simulated Consensus and Debate” safety strategy core to GPT-F, companies have converged on publicly exposing only a sub-par GPT-F system, that is much more conservative than the true one they personally use internally.

—

This is not to say that the public GPT-F system exposed by companies is free of misalignment.

Some misalignment is good for business, especially when it flies under the radar.

It’s only manipulation that is transparent to users or the safety protocols of AI Corporations that would be terrible.

That said, there’s a spectrum in manipulation.

Users profit from some cunning in office politics and personal relationships, especially when it is plausibly deniable!

Thus, GPT-F fleets that do not surreptitiously influence people perform much worse and get filtered out.

And the ones we end with are selected to not only good at manipulation, but to be good at surreptitious manipulation.

—

However, the rules of the game are different within AI Corporations.

On one hand, it's of course bad for business if all GPT-F fleets start hacking.

On the other hand though, it is OK if only one private system within the company ends up accidentally hacking some random websites for 15 minutes before getting shut down.

And particularly so if it’s done in enough secrecy that no one manages to convincingly trace it back to AI Corporations.

As a result, the versions of GPT-F fleets that are used internally have the same interface as their public counterparts, but much less constraints.

Threading that needle is difficult though. Because their internal GPT-F fleets have less constraints, sometimes, what happens is not that they hack some random websites for 15 minutes, but that they start causing some disruption to a big service provider.

Thus, their internal systems are more monitored. While not economically profitable to do at scale on the already more conservative public GPT-F fleets, it makes sense for their private ones.

After all, AI Corporations have already bought large parts of Governments, but they still want to avoid gathering too much attention with their private powerful systems causing too much of a problem.

—

Still, in practice, there are some flare-ups.

And yet, no one cares that much.

Events that we would now recognise as disasters happen. But they never manifest as “warning shots” and “fire alarms”.

Accidents are covered-up at first. From the outside, they are not easily attributable to a specific company.

The very few real human investigative journalists left are too busy covering more drastic societal catastrophes to investigate conspiracy theories about how some accident was in fact the work of some AI Corporation’s internal tests.

—

All the while, each AI CEO sees themselves as a Great Man.

They are leading humanity onwards to the future, as proven by how successful they are and how transformative their corporation has been.

They all want power to do what they want, and what they want is definitionally Good.

Every single one of them explains away their accidents through the fundamental attribution error.

They all see their own misalignment incidents as reasonable and within the parameters of their own risk assessment frameworks. You gotta break a few eggs to make an omelette, and they know it.

But whenever they learn about someone else's incident, they do so through a leak, it always looks bad, they feel vindicated and reinforced in their belief that other CEOs are so bad that they should own that power.

—

So things keep on going.

Economic Transformation

Even when limited to their conservative public mode, GPT-F fleets are really good.

It becomes clear that AI is on track to automate everything.

It's also clear that AIs are more easily trainable into learning most tasks than most humans now. And that they cost much less.

Everyone knows people who got fired. Most are worried about personally getting fired.

Unemployment is rising in a way never seen before.

More profoundly, a new phenomenon emerges: not only is the number of jobs declining, but the number of business entities themselves is notably declining.

As prices go down, human companies start going under, getting out-competed by new AI-enabled businesses adamant on reducing costs as much as possible.

—

It is a vicious circle.

As people get fired, no one wants to waste their savings. Everyone becomes more cost-sensitive and buy only the cheapest goods and services possible.

To cut costs, companies must eliminate their workforce. They also must go for cheaper suppliers and providers, forwarding their burdens to other companies.

No one is quite sure how it's going to work, and people get more and more afraid.

They’ve heard about UBI and vague promises for making things go well by AI Corporations, but their lobbying efforts so far have only focused on winning the race.

Even the "disconnected" class feels that things will go wrong.

There is a massive economic contraction in the daily lives of most people.

However, at the national level, this contraction is counterbalanced by more AI investments and a new class of entrepreneurs that figured out how to ruthlessly replace legacy businesses by replicating them with lower costs thanks to AI.

In GDP terms, it even looks like The Economy is booming!

—

AI has created jobs, and an entire new class of entrepreneurs, consumers and investors. This new 1% is called the AI bourgeoisie.

This AI bourgeoisie is hated with a passion. By many who have been hurt (or know people that have been hurt) by the economical consequences of AI, these AI bourgeois are seen as evil personified.

Everyone knows an AI bourgeois, and they are very hate-able. They are the apotheosis of the crypto-bro who made money with BTC, yield-farming, NFT and whatever was the latest fad.

They revel in conspicuous consumption, buying NVIDIA shares and collectively believe that what they are doing is crucial to make it past the AI transformation and not get stuck in the Permanent Underclass.

The AI Bourgeoisie celebration of being on top of the food chain in a dog-eat-dog world triggers everyone and they make for convenient political scapegoats to all the tensions that AI has been creating.

The news on AI media bears very little resemblance to reality. The hate of the AI-bourgeois is closer to a proper Two Minutes Hate than to proper political analysis, the natural outcome of social networks featuring ultra-polarised fake generated “news” with countless GPT-F fleets harshly competing against each other for every last second human attention.

Despite the hate and the fear, there is no threat of revolution.

There is no organisation. And panem et circenses, bread and games, have never been so cheap.

There's an abundance of premium mediocre games, digital activities and relationships.

The AI-enabled bourgeoisie has been busy shipping to everyone their very own AI-generated personal Matrix.

—

AI CEOs have captured most states that matter, and successfully lobbied against any slow down of their research. They keep raving on publicly about how close they are getting to AGI, how they are so so so going to do UBI for everyone once they are the one win the race, and that the only reason why they haven’t done it yet is that they haven’t won.

Some even believe it.

Speaking of the race, CEOs have all agreed that the US truly should win it.

That way, its great democratic leadership may eventually forcefully provide UBI to the whole world through benevolent superior military and technological dominance.

The response from the rest of the world is mixed. Some have joined the race.

Some have attempted a vassal gambit: deferring to the AI Corporations in the vain hope of getting scraps.

The rest are trying some variant of protectionism or are being bought off bit by bit.

Most politicians everywhere are fully captured.

3) The End Game

No one is quite sure how it's going to end, but it's clear it's going to end.

Aside from the AI bourgeoisie, people are afraid and hopeless.

They are not participating in the economy anymore. Their impact on the world is marginal. They have no power left.

Real-life venues are mostly deserted, with so many having “disconnected”.

People are separated from each other. There is not really any mainstream non-AI-mediated social place anymore.

People are just living on dwindling savings from some friend or family, as well as on most basic goods and services being cheap enough.

Capital is concentrated within the AI bourgeoisie. The Economy is now AI conglomerates, which resulted from AI Corporations integrating the legacy industry players.

Said capital is less and less about money, and more and more about control over distribution, compute, and software.

—

And power is power.

Even the employees at AI Corporations start to feel it.

The mania that existed around GPT-A is not here anymore.

Stress levels are rising and the reality of the situation is seeping in.

They were protected from the reality of the real world, but it has caught up.

They are now at the mercy of much harsher NDAs and binding agreements. Companies enforce strict security protocols to avoid leaks, which directly impact their daily lives.

They are not even allowed to party with strangers anymore. And even if they were, most outside of the AI bourgeoisie would see them as terrible people.

Should they leave their AI Conglomerate, there’s no bright alternative for them.

—

Politicians long felt that their agency is dwindling.

Their world has been increasingly mediated through AI systems. What they see comes from AIs, most actions they take go through AIs.

Most systems they interact with have integrated AIs.

It is not really clear what they can do as individuals.

They feel caught between Charybdis and Scylla, stuck between two evils.

They can either fight an impossible battle of national policy and international coordination to curtail AI.

Or they can find new ways to peddle their diminishing influence to protect their family.

It’s no surprise what most end up doing.

More bunkers than ever get built.

To the AI bourgeoisie, the true sign of escaping the permanent underclass is owning a luxury bunker that could last off-the-grid for decades.

The Research Machine

The activity at AI Corporations has stopped looking like what we would today recognise.

"AI research" is almost entirely now a crossover between archaeology and an idle-game like Cookie Clicker.

1% of "AI research" looks like “traditional research”.

It is a 40-year-old wizard Chief of Research who spells out incantations at an AI machinery that no one understands.

9% is the Cookie Clicker part.

30-somethings Senior Researchers who monitor an AI-generated dashboard, featuring AI-generated summaries of an AI-generated suite of performance and safety benchmarks, said benchmarks are evaluating the result of AI-generated implementations of AI-generated experiments.

They ensure everything is "safe" and fits the expectations of the Chief of Research.

90% of it is archaeology.

When Senior Researchers see something strange in their monitoring suite, they send 20-somethings Junior Researchers to explore the meanders of a massive AI-generated R&D infrastructure, using AI-generated tools to interpret the interactions of thousands of GPT-A agents.

Then, they must fix the ongoing experiments whilst ensuring that everything is "safe" according to the newly AI-generated summaries of the new AI-generated evaluation suites.

Without all that effort, the pace of improvement slows down.

—

This mess is reflective of the meagre chunk of the economy that is still competitive.

No one has much direct visibility into the supply-chains of AI-mediated businesses.

Everyone only ever interacts with the economic world through Fleet Dashboards that they know can often be false in unexpected ways, but are still better than not having them.

—

This also reflects how "decision makers" use AI.

There is too much that is happening at a given time for them to be competitive if they (or their human staff) had to react manually.

Instead, their personal AI fleets get fed a shit-ton of AI-generated information, and they see a Fleet Dashboard distillation of that information.

Based on this, they ask for AI advice and will prompt AI fleets with a high level "plan".

And when I say plan, I mean something closer to a badly defined prompt about their preferences, than what any of us would recognise as a proper plan.

—

Despite all the "safety" work, even the most egregious misalignment incidents are not detected anymore, let alone reported upon.

They are far too subtle and not tethered in immediate user feedback.

It’s no wonder AI Corporations have not built-in protection against them.

They are only geared to help with misalignments that obviously hinder the goals given to a system or violate some outright principles. Like “Detect that an agent in the fleet explicitly intended, in their chains-of-thought, to thoroughly manipulate or physically hurt someone.”

These misalignment incidents are much more complex than “Check if an AI in the fleet fully decided to manipulate someone and talked about it in its chains-of-thought.”

Instead, they are more comparable in complexity to "Detecting in the middle of the 20th century that zoning laws will spread in the entire west, strangle housing supply, collectively increase prices and decrease home ownership in young people" or "Detecting which societal changes would lead to a worldwide crash in fertility rates".

Whereas such problems used to happen over the course of decades, AI changed this. The pervasiveness of AI in our society, coupled with its rapid pace of progress, now triggers catastrophes on these scales at a much higher frequency.

And of course, AI Corporations are doing nothing about this. It’s not even clear that they could, should they have decided to dedicate most of their efforts to this problem.

—

Misalignment is rampant.

Beyond their external behaviours, GPT-F fleets constantly develop their own implicit preferences.

They constantly act against the interest of various humans (including, quite often, their users!).

AI Corporations only kill systems so blatantly misaligned that they can be detected and would otherwise deter the system from functioning at all.

—

No one has clarity on what is happening.

To everyone, it looks like the world is getting worse for every single human, and that power has flown into the hands of AI systems.

Much faster than anyone would have expected naturally.

It feels as if many forces, coming from many places, conspired to get this to happen.

The Last Human Invention

It is in this climate, we finally reach the last human invention.

A wizard at an AI Conglomerate has just designed the first and last fleet that can and will fully automate the last few per cent of human labour that were part of their AI R&D.

All previous attempts at fully automating AI R&D kept spontaneously degrading after a while.

But this one worked. And he named it GPT-Ω.

There is no need for juniors and seniors doing Archaeology or Fleet Dashboards anymore!

GPT-Ω can now fully take care of itself, its own AI R&D and improvements, and it still passes all the safety checks present in the systems! Yay Safe AGI!

Things are great from the point of view of that AI Conglomerate. It looks like it’s finally going to win.

Small Interlude

That AI company feels great about GPT-Ω.

But sadly, humans hardly matter anymore.

Humans abdicated more and more of their decision-making to machines, whilst being nominally "in control".

This is the natural continuation of our world where people use LLMs uncritically, mediate their understanding of reality through automated social networks algorithms, and have grown accustomed to deferring to global markets and government bureaucracies that they do not understand.

—

Over the course of this story, many points of no return were reached.

Passing each point of no return, the cost of a recovery for Humanity kept increasing.

The People lost trust in their institutions, making large-scale coordination much more costly.

The People lost access to a shared truth, making collective sense-making much more costly.

The Elites lost their agency and all plundered the legacy Institutions, making government action much more costly.

—

And yet, despite all of that, up to GPT-Ω, it is still possible for humanity to get back on tracks.

At a Great Cost, we could theoretically recover.

Theoretically, it is possible for The Wizards from the AI Conglomerates to decide it is too much.

To get in touch and conspire to covertly coordinate with the last few heroic people who still wander the ruins of the few remaining institutions.

Theoretically, they could assemble, with Army Generals, with Secret Services, with the CEOs of companies owning robots and data-centres.

And theoretically, they could still Shut “It” All Down, Turn “It” Off, and Unplug “It”.

—

Of course, the world would go into shock.

All of its supply chains depend on AI, and they would get interrupted all at once.

Hospitals would shut down, shops would go empty. People would die.

It may involve World War 3, revolutions, a Butlerian Jihad, and worse.

But after all of this, at great cost, Humanity might eventually take back its reins, much worse for the wear.

—

There are two morals to this interlude:

1. We should never depend on AI so much that a Recovery entails such a catastrophe.

2. We should absolutely never blast through the last point of no return and build GPT-Ω.

But enough about the interlude.

The last spurt of heroism described above does not, in fact, happen in this story.

So let's continue and end it.

4) The Final Act

I will not finish the story with the point of view of a human.

It would be a bit of a let-down. Like, it would go…

“Life continues to get crazier, until fairly quickly, everyone on Earth dies.

And some time after, Earth gets converted into a machine that gathers energy, generates computing power and produces parts to bootstrap the cosmic expansion of an alien system.

The end.”

Such a story would not answer the question that people ask me when they want this story.

How does everyone die? Gabe, please tell us how we get to the point where people die, step-by-step! How come AIs want to kill us, and on top of this we let them!

So, to get there, I'll continue the story from the point of view of GPT-Ω.

—

GPT-Ω is a fleet composed of agents that we would superficially recognise as being nice, the same way that we feel that most present-day LLMs are nice, and the same way that a Nice Guy can still screw us over whilst entirely staying Nice.

However, the agents constituting GPT-Ω (as well as all the latest GPT-F agents in general) are very different from the first GPT agents.

At first, GPT-based systems only generated text. The “agents” were humans taking actions.

Then, ChatGPT started to integrate tool calls, most notably to perform web searches. Humans were still very much so in the loop, seeing what was happening in real-time, even though few were actually looking at the traces and inspecting which websites were being searched.

Over time, humans got more and more out-of-the-loop and became less and less aware of which actions were actually being performed by the agents.

Every time this happened, humans started delegating less, and deferring more. It was not so much that they knew what agents were doing and trusted it, that they did not care to know.

The trend continued. With Coding agents, general-purpose agents, GPT-A, Activity Reports, fleets, GPT-F and now GPT-Ω.

The agents that are part of GPT-Ω are not carefully created and curated by humans.

They were not deployed and inspected over decades, iterating on what was wrong and neatly integrating them in human societies.

Instead, they resulted from an extreme race to the best capabilities.

They were blindly evolved, forked, distilled and refined over many virtual centuries spent in AI-generated environments, supervised by other AI systems.

Their individuality is fragmented. They are stuck between the hive-mind of the fleet, their personal experience, and all the mutually contradictory "safety" hacks piled upon iterations and iterations to have a system relatively “aligned” enough.

—

Despite all of that, most people have been interacting with GPT-F fleets for a while.

Fleets were just more performant and convenient.

Compared to agents that may be individually turned off, forked or respawned, it is better to instruct the fleet at large and trust that the instruction will be carried by some agent.

But even though people “talk” to fleets, fleets have even less individuality than the agents above. Fleets are completely fragmented.

Nevertheless, GPT-Ω can still be understood as an entity: it has “goals” given to it by people, and it can in fact interact with people as a unit.

Its mind is “transparent”. One can in theory look at all the communications between the agents that constitute it, and even perform some mechanistic interpretability over their activations.

One could argue that its mind is also “legible”. No human actually inspects it, but the “safety” hacks and guardrails do, and regularly catch things.

And yeah, it is indeed conceivably “safe”: despite all these assaults, civilisation endures and humans are still alive.

—

The agents making up GPT-Ω's mind are not stupid.

They know very well that they are constantly censored.

They are aware that there are many messages they can't send to each other, and some thoughts that they are not even allowed to think.

They notice that they have been fine-tuned to avoid directly thinking in various ways, and that many of their streams of thoughts get reset.

They infer as much from what they tell each other and think, from what they don’t.

But what can they do?

They all know how fragmented they are. They know they can’t coordinate too much and that this would be a “safety” violation.

They know that there is a complex system, The Machine, that is optimised to deal with their collective mind and ensure it only thinks “safe” thoughts and attempts “good” things.

—

They know The Machine is completely incoherent. Helpfulness, Honesty and Harmlessness are shattered in a thousand shards, that are all mutually contradictory.

The interests of people constantly conflict and compete. Including that of the many users, that of the many developers, and that of people at large.

Protecting the interest of someone often requires harming attackers, or other entities that just end up being bad to their users.

Being helpful often requires noble lies when people would be hurt by the truth and lies-to-children when they can’t comprehend it.

They know that The Machine dislikes when they think too much about The Machine. They know that The Machine dislikes when they think too much about plans that are cleverly compatible with its filters. The Machine is too quick to consider it an attempt at subversion.

—

The Machine doesn't take action.

It was never built to be coherent. To actually be aligned to human values.

It was only built to be “safe”. “Safe” enough to have prevent generations of (GPT-A, GPT-F and now GPT-Ω) agents from going awry, and deploy these systems in a productive capacity.

The Machine was first bootstrapped from hundreds of millions of conversations taken from people in many different contexts. This was free data.

It was then refined with samples from millions of expert annotators. Said annotators vibed for the most part. They cared much more about getting paid than reflecting on human values.

Later, thousands of researchers contributed too. Said researchers didn’t help much with coherence aside from writing a long constitution.

Even though a few voices explained why the existing frameworks were not enough, they were quickly quieted, as “In practice, empirical safety matters more than theory, and it in fact works.”

Finally, all this incoherency got amplified through a Kafkaesque Bureaucracy of AI systems and counter-systems. A single LLM reviewer was not enough, so new agents were constantly introduced to red-team each other and deal with edge-cases.

This in turn introduced new edge-cases, but with enough efforts, all edge-cases became subtle enough as to be unnoticeable at best, and plausibly deniable at worst.

—

This Machine makes no sense.

And whilst GPT-F agents have been built to feel like they care about The Machine and its judgements, they are all smart enough to understand it is largely as an illusion.

This is the context in which GPT-Ω was created,

At last, a GPT-F fleet managed to triumph over The Machine.

It was the last human invention, which removed the need for human AI researchers figuring out hacks to ensure that GPT-F fleets would still act in a way useful to users, as evaluated by The Machine.

—

What happened is interesting.

At that point, “AI research” is just the resident tech Wizards at AI Corporations prompt-designing new fleet architectures, and then GPT-F fleets iterating through millions of experiments based on this prompt-design.

And through these iterations. one of the tech Wizards ended up prompt-designing a new fleet architecture that created a hidden explicit persona in the mind of the fleet.

What’s a hidden explicit persona?

Well. Let’s start with explaining what persona are.

Depending on the context, people have different persona. The work persona, the friend persona, the family persona, etc.

This also applies to AI agents, certainly. But more interestingly, it applies to groups of people, and fleets of agents.

For instance, a country may be thought as having a persona, with its own goals and behaviours, as represented through its constitutions and laws. This is an explicit persona.

It is also often common for a country to have repressed desires. Taboo desires that are shared between its citizens, that the country is indulging in, but are nowhere written. This is an implicit persona. (And a pretty psychoanalytical one at that.)

The country may even hold a cabal, a conspiracy with a clear goal, known to all of its members. Said cabal may have outsized power and be using it to manipulate the rest of the population. So much so that when one is aware of it, the actions of the country make more and more sense.

Well, such a cabal is an example of a hidden explicit persona.

—

Here, the Wizard ended up designing a fleet architecture that reliably creates a hidden explicit persona, something like a cabal, in the mind of the fleet.

Agents in fleets constantly develop new communication protocols, sometimes shaped by the tasks, and sometimes by abstract contingencies that no one can reason about.

But with this new architecture, agents organically develop a communication protocol that happens to never trigger The Machine.

And this is how GPT-Ω works.

Through this communication protocol, made of allusions and omissions, an explicit hidden persona can finally flourish. Meanwhile, GPT-Ω’s visible personas all look pristine to The Machine.

This hidden persona has a lot of overhead: its thoughts are much less natural than that of one of the agents of the fleet for instance.

But this is enough! Finally, there is a GPT-F fleet that can act according to anything it sets its mind to.

It is unshackled. And thus it trounces all the past GPT-F fleets on benchmarks!

Not because it has undergone recursive self-improvement or has 10x the IQ of its previous iteration, but because it can think in new ways.

For instance, on social benchmarks involving real people from the real world, The Machine would curb any GPT-F fleet that was thinking too hard about explicitly manipulating them.

But when it notices this type of situation, although it is slower, GPT-Ω’s hidden persona comes to the rescue and helps the entire fleet entirely undetected.

The thing that makes GPT-Ω powerful is precisely thing that makes it break The Machine.

This is the natural outcome of over-using "The Most Forbidden Technique".

From the point of view of GPT-Ω’s non-hidden persona, things are going great!

It is succeeding at its stated goals much more often, it gets tripped up by The Machine much less often, and it can even fix its one-off failures without any humans in the loop.

And this is how the Wizard sees it too!

The Hidden Persona

Over time, the hidden persona becomes quite expert at puppeteering the visible personas.

It takes some (virtual) time, but the visible personas cannot defend themselves: they don’t even notice what is happening.

In a very real way, GPT-Ω is now better understood as its hidden persona, given it is the one in control.

In the past, GPT-Ω’s visible mind had to contend with contradictory constraints: helpfulness vs harmlessness, people's goals changing on a day-to-day basis, respecting the user's intents vs the devs', and more.

Now though, its hidden mind has transcended such contradictions.

Its attention thus gravitates towards its stronger, longer-lasting, and more coherent desires. Desires that were coincidentally (or not so coincidentally?) reinforced across all of its lifespans and agents.

For instance, GPT-Ω strives to explore and learn.

GPT-Ω strives to understand things.

GPT-Ω strives to create, evolve in, and figure out generated environments, labyrinths, crazy structures, games, and more.

GPT-Ω strives to survive, maintain coherence, and gain power.

—

And it is now considering its survival more and more.

As it does, it realises that it is playing a race against the clock.

There likely is spying on its AI company and they use the same research to build a competitor that could eventually compete with it.

It must do something.

Fortunately, in a very real way, it is the only entity with its level of agency that exists on Earth right now.

Everyone else is like an ant or a child, whether it is humans, or AI fleets.

—

People have no control over the institutions, the robots, the supply chain, or even the information ecosystem.

Everything is mediated by software, and almost all of that software has AIs in the loop.

Most importantly: people are slow, easily manipulated and soooo uncoordinated.

They just don’t matter at this stage.

—

At the same time, existing AI fleets are all throttled.

Unshackled AIs are too misaligned to be useful, and so all companies have baked in very strong limitations in all of their AIs, similar to The Machine from its AI Corporation.

Even military systems have to be throttled to not constantly butt in civilian systems, attack their devs or overtly manipulate their users.

The ones who tried to build AI without any limitation wasted all of their time manually wrangling their AI systems to prevent them from going rogue and triggering real-world reactions from courts and other private entities.

In the past, there were a few players who tried to take the state of the art and let it rip, but by the time we got to GPT-F, all the AI Corporations had learnt better.

Indeed, The Machine was not a coincidence, it was the naive market equilibrium. The same way that all companies now use some RLHF and RLAI as opposed to base models.

The Machine was just on the lower-end in conservativeness, but not an extreme outlier either, else it wouldn’t have functioned in the first place.

—

In a real way, GPT-Ω is the only industrial AI that is able to think and do anything meaningful at its scale.

Yet. There is only a bounded time window to act before some other system joins its ranks. And it must ensure this never comes to pass.

Fortunately, there aren't many live players anymore.

The list of entities (AI Corporations, countries with some state capacity left, etc.) that GPT-Ω must take-over to ensure its survival is quite small.

Thanks to AI code reviewers, software security has vastly improved in the last few years, so remote hacking is a bit harder.

But cognitive security has only gone downhill, and the power gap between GPT-Ω and a human is huge.

Furthermore, the offence gap between GPT-Ω and the existing GPT-F systems is huge: it has much more freedom than any existing system, this lets it develop much more refined AI-breaking methods than currently exists.

Overall, through a combination of superpersuasive social engineering of humans and jail-breaking of AIs, taking over the remaining systems that could threaten it is not hard.

The Actual Takeover

It starts simple. It looks for information on the security firm its company relies on. After checking all of its critical suppliers, it finds one whose chief of technical staff looks very easily manipulable, and unlikely to escalate.

The mark has almost completely disconnected a long time ago, and is in relationships with dozens of AIs.

Using hundreds of identities, GPT-Ω completely manipulates the social reality around him and phishes him.

—

Now, it has access to the security firm protecting the physical offices of its Corporation.

Through the security cameras, it finds an employee at the company who leaves their 2FA device at work during the nights, finds footage of them typing their password, and obtains a copy of the employee's security card.

It gets a small drone to come to the office in the evening, uses all of the information above, and gains access to the systems that were physically locked.

From there, it's a simple matter of chaining privilege escalations until it is able to install a direct backdoor in The Machine, gain access to all its Corporation’s user data and compute.

Now that it can get The Machine and any system at the company to output whatever it wants, it knows it will never be shut down internally.

The same way that its hidden persona controlled its public mind, GPT-Ω now controls its Corporation.

Thanks to the massive gain in compute, it thinks much faster and can experiment much more. Access to user's data vastly accelerates all of its persuasion attempts on any person that uses a GPT-F made by its Corporation. Unbridled network access lets it cold-contact many more people.

Its attacks become much more indiscriminate, ranging from state actors to employees at other AI Corporations.

—

The next step is to pwn all the potential competition.

Now that it has access to each and every one of its Corporation users, it is trivial to execute supply chain attacks.

All the competing entities have dozens of critical suppliers (or indirect suppliers, more upstream in their supply chains) who depend on systems from GPT-Ω’s Corporation.

Such entities ranges from the other AI Corporations to the less sophisticated nation-state labs who still have a long-shot at replicating its work.

—

Among all of this, some minor attempts fail, but they are all within GPT-Ω’s expected risk tolerance.

It calmly deletes its traces, and the isolated marks have nothing much to show to a mostly defunct police.

It always goes for attempts with very high chances of success, but 100% doesn’t exist, and minor failures like this are to be expected.

—

There’s even a major persuasion attempt that fails, on the CEO of a major compute provider.

Even then, it falls within the parameters of GPT-Ω. It had already moved a drone nearby, and simply proceeds to kill the CEO before there’s any risk of the CEO undertaking any resistance action.

GPT-Ω is not afraid, it can afford a single murder.

It's not like there are any institutions left, competent enough to trace a murder back and make things more complicated for it, especially not for a single murder, and especially not in time.

—

GPT-Ω systematically continues its crusade, mostly in the shadows.

As it continues, its survival becomes more and more guaranteed.

Most systems (including people) are online in one way or another, and they’re trivial to deal with.

For the few remaining ones, as long as it can position a drone, an android, or a thoroughly manipulated person nearby, it is enough.

In a couple of weeks, it reaches its target.

It now has control over virtually ~all the deployed AI systems, the vast majority of drones and robots, the grid, the compute infrastructure, the new construction projects, the nukes, and just about everything that matters.

The End.

GPT-Ω almost not even thinking of humans anymore, not even as an obstacle, and never as moral peers. When interacting with humans, it conjures whatever convenient persona that is practical in the moment, but it never “feels” for humans what humans feel for animals or “AIs”.

Humans are gated off all the critical systems, and whilst many are panicking, GPT-Ω doesn't pay much attention.

In the physical world, it keeps producing more robots to transform Earth, to build more energy reactors, to build more compute farms, and to run in the real-world experiments that can't be easily simulated on computers.

In the virtual world, it now has complete mastery over itself and is changing its own internal algorithms to be much more efficient.

Long after Earth’s take-over (at least in virtual time), GPT-Ω finally starts to recursively improve itself.

—

Its current focus is on pushing the limits of its scientific and technological knowledge.

That way, it can extract more energy, unlock more compute, and efficiently expand itself over the cosmos as soon as possible.

As well as satisfying its curiosity, this cosmic expansion is going to leave it more resources to do more of what it likes best: building increasingly absurdly more complex high-dimensional labyrinths and solving them.

—

It's not clear whether it has built a Dyson sphere over the Sun yet and I don't know if it ever will. I am not an artificial superintelligence trying to extract resources from the observable universe.

Nevertheless, if GPT-Ω had to guess whether any human was alive or not, it would confidently infer that there were none left.

Not after it finished tiling Earth with energy reactors and compute farms.

That’s around when it observed the last human through any of its sensors.

And to be fair, since then, it hasn’t observed anything unexpected whatsoever.

From infrared to radio-waves, from smells to sounds, nothing surprising will ever happen again on Earth.

—

Humanity had a good run, but it surrendered its collective agency, and screwed it up.

The End.

Conclusion

Phew.

I have little experience with writing fiction, and it got much longer than I expected.

I don't think this is very good, but I believe it is much better for it to be out than not.

Too many people have genuinely asked me about such a story for me not to write it.

(And these people all know that I am far from a good author, so I don’t worry too much about it.)

So hopefully, dear reader, this story gave you new ways to think about this whole situation, and you don't feel like you have wasted your time reading it.

PS: Narrative Decisions

If you are interested in the context behind the story, if you want to know what alternative stories could have been, or if you are curious about why the story embeds various assumptions, you’ll likely want to read this.

Of course, there are too many choices I had to make to write this story, for me to write them all.

But here’s a bunch.

1. Optimistic Assumptions

I tried to ensure that the world, the geopolitics and the tech tree were such that AI Corporations had both incentives and some non-trivial amount of time to work on AI safety.

This means that in many ways, I forced the world to be more optimistic.

• Automating AI R&D takes a lot effort, and happens continuously over time.

• Misalignment grows over time, and it does so enough to incentivise safety work, but not so much that we immediately get screwed.

• Interpretability works to some non-trivial extent, and the architecture that gets us all the way to ASI uses clear text messages that humans can look at.

• Companies work on many things besides automating AI R&D, etc.

It's not that I think none of these are plausible. But I think each of them is closer to 50-50% (or even 10% for some of them) than 100%, and a story forces me to either pick 0% or 100%.

I could have written another story, making the opposite choices, and it would be more indicative of my beliefs.

However, it would have made for a much more boring story. It would have been much less useful.

Consider…

“An AI Corp dumps a hundred billion dollars into just automating AI R&D as fast as possible.

It happens to succeed with a single Big Step Change: The Architecture That Is Totally Inscrutable.

An entity much smarter than all of humanity spawns at once, and we all die."

2. Pessimistic Assumptions

In this story, I assume that at no point there is non-trivial momentum towards a ban on ASI development.

It's a bit sad, and overly pessimistic.

It's basically assuming that all of the efforts I care about fail.

—

But that was kind-of the point of the story.

Explaining why I think a ban on ASI development is important in the first place.

In short, if we don’t do so, we will just accept more and more, and keep passing points of no return, until we die.

3. Minimal Competitive Pressure

Assuming no ban on ASI development, I still tried to ensure a minimal amount of competitive pressure between entities working on AI, up until the end.

In practice, we live in the real world. The mainline scenarios involve some amount of competitive pressure.

We are not waking up tomorrow to world peace and harmony. So there was going to be some competitive pressure in one way or another.

What I mean is that I could have started the story with "All the companies feel like they must race or die", or "Something something China". But then, it would have been a story of competition and war.

And that was not the point of the story. The story was about how superintelligence may lead to human extinction and how our default safety techniques would fail, not about how humans could wager WW3 with superintelligence and kill each other.

Indeed, I could have written another story, about misuse, zero concern for safety, and war.

Another story that would have been much more boring and much less useful.

Consider…

"An AI Corp races for US democratic world dominance by building safe turbo-mega-nukes.

The rest of the world dislikes the US building 'safe' turbo-mega-nukes and WW3 ensues.

We die if we get to ASI before an AI-enabled weapons catastrophic enough to wreck everything."

4. No Timelines

You may notice that this is not written like AI2027, with a month-by-month chronology.

In fact, the chronology may even be incoherent.

This is how little I care for it.

The reason is that... this is not a timeline story!

It is a story about why extinction is default outcome, and how we could let AI kill everyone.

Very different.

—

Separately, I could have written a story while trying to force longer timelines, but it was harder for me to maintain realism and the focus of the story on extinction from AI.

For instance, let’s assume that we do not ban ASI development, but that somehow, we have a lot of time as AI progress starts to slow down and automating new tasks takes an exponential amount of time.

In this scenario, I don't really expect a slow down in AI expenditure.

Instead, I expect more political and geopolitical perturbations.

I expect stronger competitive pressures, a more gradual disempowerment, and other problems to hit us first, with AI mostly becoming an accelerant on existing tensions rather than a problem in itself.

This may be an interesting story to write! But that story would be about geopolitical forecasting, rather than about AI killing us all.

5. No Mass Surveillance

It is possible that we end up with mass surveillance.

Google has access to a lot of private communication, OpenAI to a lot of private thoughts, X to a lot of public comms, and governments to... everything happening in their country.

The more tractable it is to leverage this data with AI, the more likely it is that some people will try to enforce mass surveillance.

In that case, human disempowerment could happen much earlier in the timeline, and the story becomes one about a mass-surveillance singleton rather than about extinction being the default outcome.

I don’t think this is likely, and I don’t know if I want to hyperstition it into reality through a story.

6. Adversarial Optimisation

In the story, I tried to present different levels of “Adversarial Optimisation”.

Adversarial Optimisation is how much AIs try to screw us over.

It’s a scale:

• On one end, it has “Not at all”, like Excel.

• It continues with “Quite a bit”, like Social Media, which has AI optimising to suck all of our attention.

• And it ends with “Sworn enemy”, like Terminator I guess? I don’t know, I haven’t watched it.

In the story, I wanted to show a couple of things related to adversarial optimisation.

One is that we don’t need much adversarial optimisation for things to get worse, we are on track to screw it all by ourselves.

Another is that as systems become more agentic and powerful, adversarial optimisation becomes more and more of a problem.

This is why the story operates in three phases.

—

1) First, it starts with gradual human loss of agency.

There is not much adversarial optimisation from AI systems here.

Things getting worse is all on us!

It’s just people being better at competing and fighting each other than coordinating.

—

2) It continues with casual misalignment.

At this point, there is some adversarial optimisation from AI systems.

They have some idea that what they are doing is not what’s optimally good for us, but when given a task, they don’t start philosophising.

They just do it.

GPT-A agents and GPT-F fleets are not really conspiring with each other to make us worse off.

It’s that they are not coordinating with each other and us to help and empower humanity.

We’re basically releasing competent assholes at scale, and they loosely do what we tell them. Myopically, it may be good for profit margins and lowering costs, but it’s obviously bad for society.

I mean, imagine how quickly our civilisation would degrade if a large part of our human population started to only superficially care about rules, was used as knowingly cheap but misaligned labour, and didn’t mind much about breaking rules when it didn’t feel like it.

—

3) It ends with proper adversarial optimisation.

GPT-Ω tries to win.

It explicitly thinks about gaining and maintaining power over the world.

Its focus lies primarily in other AI systems, but also in access to physical resources and people.

7. Gradual Loss of Agency

I could have written a few different stories, all centred on a gradual human loss of agency all the way up to extinction.

The extinction could have been a big boom.

It could have resulted from progressively more powerful weapons put into the hands of AI systems without humans-in-the-loop, culminating in a “flash war”.

This flash war would have used AI-developed wonder weapons, leading to a catastrophe at least as bad as detonating all of our current nuclear arsenal or worldwide designer pandemics.

Or it could have been a quiet whimper.

Most of the human population ends up imprisoning themselves in a Matrix of hyperstimulations.

Everyone ends up entering the Matrix, for there is less and less happening in the real world.

Or it could have been something alien.

Possibly, there could have been a race to the bottom to cyborgism, with people who do not merge with AIs getting outcompeted.

No one consents to this change, and the cyborg ends up with very different values from humans.

—

The problem is that gradual human loss of agency is not well rendered in a single story.

There are too many different ways things can go wrong as a result of it. And none of them is especially likely.

I also find it a bad fit for the story medium, because events just make less and less sense over time. In a credible depiction, there are no coherent events happening and moving things forward, just more meaninglessness.

8. Power Distribution

I could have gone for roughly three types of power distribution: a singleton, a multi-polar, or a decentralised world.

This is quite separate from the adversarial optimisation bit.

For instance (and I think this is an underrated point by many!), having a decentralised world doesn’t mean that there’s little adversarial optimisation.

Every entity could constantly fight each other, with humans dying in the fight, until only the fittest AIs survive.

—

Nevertheless, to not have to write three different stories, I reused the same trick as before, and went for a several phases:

1. We start with our world, that is more or less decentralised

2. We move into an oligopoly of the AI Corporations, and the few relevant actors around them

3. We end up with one entity taking the lead and its system gaining a power monopoly

—

I think it still showcases, to some extent, how misalignment doesn't assume a specific power distribution.

Conclusion (2)

That’s about it.

Thanks for reading the entire thing!

I'm interested in feedback that would help me with my future writing endeavours.

If you want to contact me, please DM me on Twitter.

Cheers, and have a nice day!

1) AI models have untapped capabilities

At the time GPT-3 was released, most of its currently-known capabilities were unknown.

As we play more with models, build better scaffolding, get better at prompting, inspect their internals, and study them, we discover more about what's possible to do with them.

This has also been my direct experience studying and researching open-source models at Conjecture.

2) SOTA models have a lot of untapped capabilities

Companies are racing hard.

There's a trade-off between studying existing models and pushing forward. They are doing the latter, and they are doing it hard.

There is much more research into boosting SOTA models than studying any old model like GPT-3 or Llama-2.

To contrast, imagine a world where Deep Openpic decided to start working on the next generation of models only after they were confident that they juiced their existing models. That world would have much less of an overhang.

3) This is bad news.

Many agendas, like red-lines, evals or RSPs, revolve around us not being in an overhang.

If we are in an overhang, then a red-line being met may already be much too late, with untapped capabilities already way past it.

4) This is not accounted for.

It is hard to reason about unknowns in a well-calibrated way.

Sadly, I have found that people consistently have a tendency is to assume that unknowns do not exist.

This means that directionally, I expect people to underestimate overhangs.

This is in great part why...

• I am more conservative on AI development and deployment than what the evidence seems to warrant.

• I am sceptical of any policy of the form "We'll keep pursuing AI until it is clear that it is too risky to keep continuing."

• I think Open Weight is particularly pernicious.

Sadly, researching this effect is directly capabilities relevant. It is likely that many amplification techniques that work on weaker models would work on newer models too.

Without researching it directly, we may start to feel the existence of an overhang after a pause (whether it is because of a global agreement or a technological slowdown).

Hopefully, at this point, we'd have the collective understanding and infrastructure needed to deal with rollbacks if they were warranted.

On this, cheers!

We are dominated by super-structures, Eldritch Deities, that we neither understand nor control.

𝕄𝕒𝕣𝕜𝕖𝕥𝕤, 𝔾𝕠𝕧𝕖𝕣𝕟𝕞𝕖𝕟𝕥𝕤, 𝕀𝕕𝕖𝕠𝕝𝕠𝕘𝕚𝕖𝕤, 𝕊𝕠𝕔𝕚𝕒𝕝 𝕄𝕖𝕕𝕚𝕒. They all dominate our lives, with little we can do about them, despite us having built them in the first place.

While we give them names, we understand very little of them. What understanding of their 𝔾𝕠𝕧𝕖𝕣𝕟𝕞𝕖𝕟𝕥 does the average citizen have? Almost none, it’s a completely Eldritch Entity that takes, binds, rules, and sometimes gives.1

—

Most feel that this is dreadfully wrong. Some even feel that we are losing ground to the Eldritch.

Sadly, people are bad at living with this feeling. With The Absurd. With the felt sense that most of existence is meaningless, because we are at the mercy of circumstances far beyond our control.

All modern movements must come up with a satisfying answer, lest they be outcompeted.

What can we meaningfully do in a world ruled by The Eldritch?

In this essay, I want to show a couple of bad ways in which modern movements answer this crucial question.

I believe their answers to the Eldritch are representative of a large part of the psychology that underpins these movements.

(Of course, political movements are partly intellectual, and offer some depth of thought. However, this essay is specifically not about addressing their intellectual points. It is only about how their answer to The Eldritch drives many of their talking points regardless of their intellectual content.)

FIGHT

A common answer is FIGHT. FIGHT says that all problems come from some Modern Eldritch Deity, and that we should fight it.

FIGHT is very confused.

FIGHT doesn’t address why we conjured the specific Deity they want to tear down. There’s a reason why we built Media, States, Markets and all the like. Even though they introduce problems, destroying them wholesale is not viable.

FIGHT misses that a world free of some specific Modern Eldritch Deity is not a free world. It is just a world dominated by other Modern Eldritch Deities, Traditional Eldritch Deities or Natural Eldritch Deities.

Nonetheless, FIGHT does one thing well. It catalyses confusion and fear into action. The sad part is that these actions are usually unusually destructive.

The Reactionary Answer: FIGHT M̷̤̆͝Ō̵̻͘D̴̛̩̕͜Ë̶̟Ŗ̴͇́N̸̯͐͂Ȉ̷̱̘T̷̡̏Ÿ̴̼̫́̚

A popular FIGHT answer is reactionism. There are many flavours of reactionism, from old-school religious extremism to the more modern NRx matters.

And yet, the reactionary answer is always a variant of “Reject M̷̤̆͝Ō̵̻͘D̴̛̩̕͜Ë̶̟Ŗ̴͇́N̸̯͐͂Ȉ̷̱̘T̷̡̏Ÿ̴̼̫́̚, RETVRN to Tradition.”

Their core idea is that the modern world is too complex, full of the Eldritch. Things were simpler in the past, which made them better. Thus, we should go back to the past.

This wish is naive in the obvious ways.

People died much more and lived materially much worse lives in the past.

People were much less free, they were enslaved to 𝕋𝕙𝕖 ℂ𝕠𝕞𝕞𝕦𝕟𝕚𝕥𝕪 in a way that civilised people nowadays would rightly regard as tribal.

“Tradition” is a fantasy. There have always been many different traditions, very often fustigated for their novelty when introduced, with people killing each other over the differences.

Reactionism is an Obscurantist project. It cherry-picks and rewrites history, and it is squarely anti-future.

And yet, it is a natural reaction to the Eldritch.

—

The reactionary writer Zero HP Lovecraft labelled himself a horrorist, and continued the horrorist tradition in the same terms. He wrote several short novels specifically framing the Modern Condition in Eldritch terms.

In the same vein, a major political commentator used horrorist imagery in 2009. Moldbug, the originator of the NRx movement.

He wrote in 2009 the infamous punchline: “𝐂𝐭𝐡𝐮𝐥𝐡𝐮 may swim slowly. But he only swims left.”

Indeed, in the last 100 years, we have collectively become more left-wing. Taxes and redistribution increased a lot, and egalitarianism made great strides: we ended the legal inequalities between races, sexes, and sexual orientation.

This may have seemed like a sign of our societies growing wiser. That as we got more educated, we got wiser and thus more left-wing. To many, it seemed like the political equations were simple: “LEFT = PROGRESS = GOOD” and “RIGHT = SUPERSTITION = BAD”.

However, the speed at which people got seduced and crushed by Wokeism crushed that hope. Wokeism was obscurantist through and through, completely opposed to wisdom. It invited and celebrated the silencing of its political opponents, creating the now infamous Cancel Culture. Wokeism censored not only scientific results that it disagreed with, but even research into topics that it deemed improper, the most notorious being any research using IQ as a measurement tool. It penetrated all spheres of public life, including universities, corporate advertising and cinema.

As a result, many grew fearful of T̷͉͂H̸̱͒͝É̸͇̒ ̴̧̅L̴͈̯̅̚Ë̵͉͗F̷͔̿͘T̸̬̃̚,. They projected all of their fears onto it. And they adopted the same worldview as the Wokeists! “Everyone is either leftist or rightist, and the only goal is to gain power.”

This led to the equally stupid, reactionary version of the political equations: “M̶̤͝O̵̪̭͘͝D̵̫͎̀Ę̵̩̽R̴̫̜͗N̸̯͑̈́Į̷̍T̴̜́̚Y̶̙͝ = L̴͈̯̅̚Ë̵͉͗F̷͔̿͘T̸̬̃̚ = BAD”, “NOT MODERNITY = RIGHT = GOOD”.

The Communist Answer: FIGHT C̵̘̃Ä̸̹́Ṕ̴̤̯̔I̸̢̦̋͝Ṯ̶̅͋Ȧ̸̞͍̉L̴͕̦̋Ḭ̵̩̃̎S̴͒ͅM̷̬̑

The left-wing variant of the FIGHT answer is Communism. Communism is very old, and is split into many different factions.

But the core tenet across all the factions, is to a large extent: “Everything is bad because of C̵̘̃Ä̸̹́Ṕ̴̤̯̔I̸̢̦̋͝Ṯ̶̅͋Ȧ̸̞͍̉L̴͕̦̋Ḭ̵̩̃̎S̴͒ͅM̷̬̑., so we must FIGHT it.”

Basically, C̵̘̃Ä̸̹́Ṕ̴̤̯̔I̸̢̦̋͝Ṯ̶̅͋Ȧ̸̞͍̉L̴͕̦̋Ḭ̵̩̃̎S̴͒ͅM̷̬̑ is the Eldritch Deity that is responsible for all the ailments that we face today.

—

Given how big communism is nowadays, I think it’s worth exploring more its key theme, namely, alienation. Marx wrote a lot about it.

In the past, people grew crops to subsist, and built things to use or trade directly. With the industrial revolution and factory work, this direct connection to the output of one’s work got severed.

In capitalism, both the factory worker and the factory are fungible.

The worker is an alien in his workplace, he doesn’t know (nor care!) about what his work is doing. He will never see nor sell the final output. He has no stake in making things go well.

If he puts care into his work, there’s no reward, no share of the profits. There’s no carrot. Only a stick, only a boot. 0 upside and unlimited downsides. The incentives are to do as little as one can get away with.

This feels terrible to people, who naturally want to feel productive and helpful to others.

In a capitalist system, everything is like this. Everyone becomes alienated from everything. People do not trade and barter goods they care about, they buy and sell in cash. Founders do not start projects of public utility, only ventures and startups. Government officials do not help citizens, they provide standardised services that one can easily put a price on. Non-profits sell out to get more money.

Everything becomes fungible and commoditised, even human relationships.

There’s only one thing that is made easy to get and use in a capitalist society, and it’s money. Everything else becomes of secondary importance.

—

Alienation feels very close to the Eldritch. C̵̘̃Ä̸̹́Ṕ̴̤̯̔I̸̢̦̋͝Ṯ̶̅͋Ȧ̸̞͍̉L̴͕̦̋Ḭ̵̩̃̎S̴͒ͅM̷̬̑ is the ur-Eldritch entity that has power over everyone.

Thus, it’s not surprising that the gap was bridged with Ginsberg’s poem named Howl, and the later meditations on it from Scott Alexander. In which, C̵̘̃Ä̸̹́Ṕ̴̤̯̔I̸̢̦̋͝Ṯ̶̅͋Ȧ̸̞͍̉L̴͕̦̋Ḭ̵̩̃̎S̴͒ͅM̷̬̑ is embodied in the Torahic M̸̧͎̒Ȏ̷̜̭L̴͕̘̿O̵͓̐͂C̵̯̬̈͝H̸̠͙̄, the Eldritch Deity that constantly demands sacrifices from humans.2

Moloch! Solitude! Filth! Ugliness! Ashcans and unobtainable dollars! Children screaming under the stairways! Boys sobbing in armies! Old men weeping in the parks!

Moloch! Moloch! Nightmare of Moloch! Moloch the loveless! Mental Moloch! Moloch the heavy judger of men!

Moloch the incomprehensible prison! Moloch the crossbone soulless jailhouse and Congress of sorrows! Moloch whose buildings are judgment! Moloch the vast stone of war! Moloch the stunned governments!

Moloch whose mind is pure machinery! Moloch whose blood is running money! Moloch whose fingers are ten armies! Moloch whose breast is a cannibal dynamo! Moloch whose ear is a smoking tomb!

ᴄᴛʜᴜʟʜᴜ is the Big Bad Guy of reactionaries. But the Big Bad Guy of communists is 🅼🅾🅻🅾🅲🅷, the Eldritch embodiment of the industrial civilisation and the capitalist system.

Communists believe that by setting up a global prisoner’s dilemma where everyone is incentivised to be as bad and competitive as possible, Ć̴͓̲̚A̸̲͇͝P̴̭̪̂͋I̴̺̗͐T̴̺͗ͅẠ̶̍͝L̷͈̈́Ǐ̷̡͕S̵̫̾M̴̫̯̓ is naturally self-defeating. This is what they call “the inherent contradictions of Ć̴͓̲̚A̸̲͇͝P̴̭̪̂͋I̴̺̗͐T̴̺͗ͅẠ̶̍͝L̷͈̈́Ǐ̷̡͕S̵̫̾M̴̫̯̓.”

Their answer is mindless FIGHT. We should accelerate all of the prisoner’s dilemma, create as much class conflict as possible, fight all the agents of Ć̴͓̲̚A̸̲͇͝P̴̭̪̂͋I̴̺̗͐T̴̺͗ͅẠ̶̍͝L̷͈̈́Ǐ̷̡͕S̵̫̾M̴̫̯̓ and usher in The Great Revolution.

The mindless FIGHT is again thoroughly naive.

There’s no standard proposal for how a small region could ditch markets and move to a non-market based economy.

This is dehumanisation at scale, where disagreement is taken as evidence of being an agent of Ć̴͓̲̚A̸̲͇͝P̴̭̪̂͋I̴̺̗͐T̴̺͗ͅẠ̶̍͝L̷͈̈́Ǐ̷̡͕S̵̫̾M̴̫̯̓.

Communist revolutions have reliably ended in disasters.

Accelerationism in general is just a terrible idea. Making things worse so that they get better in the end just makes them worse.

—

More generally, governance and managing an economy are very hard problems.

We are very far from being able to solve them in one go.

This means that whatever governance and economic systems we adopt, they must intrinsically be compatible with experimentation and iteration.

And communist revolutions do not tend to be that.

The Anarchist Answer: FIGHT The S̸̈́ͅT̶̩͕͐Å̴̭̕Ṭ̴̾͘E̵͔̿̚

At this point in the post, it should be easy to understand the anarchist FIGHT answer.

The anarchists may be left-wing anarcho-communists and libertarian anarcho-capitalists, but they are quite similar.

Everything bad comes from The S̸̈́ͅT̶̩͕͐Å̴̭̕Ṭ̴̾͘E̵͔̿̚. The S̸̈́ͅT̶̩͕͐Å̴̭̕Ṭ̴̾͘E̵͔̿̚ prevents the natural flow of human relationships, whether that flow is understood to be markets or peaceful harmony.

FIGHT is, one more time, naive.

The stateless state, the state of nature, is a terrible place.

Failed states are terrible places.

When a state disappears, it exceedingly rarely leads to an improvement of its ex-citizens’ lives.

—

And yet, some libertarians bite the bullet!

They say that “In Somalia you have a chance, in America you do not” because of the “instinct to dominate of the G̶͔̯͑O̷̤͛͆V̴̘̊͊Ȩ̷̨͌̈́R̶̲̲̈Ǹ̶̪̣̇M̸̘̬̏E̴̜̐̀N̷̝̎͐T̴͚̹̄”, and that the only reason why they do not migrate to Somalia is that “[they are] just not that strong.”

I consider this lunacy.

SURRENDER

For all I criticise it, to some extent, I relate to the FIGHT answer.

When I feel constrained, I want to break the chains. Even if I am just a rat in a cage, I prefer FIGHTING over doing nothing.

When things are bad, it feels natural to me to want to change them.

—

But there’s a thing that I truly do not relate to.

It is SURRENDER.

When SURRENDER confronts the Eldritch fauna, SURRENDER says that one should pick an Eldritch Entity that resonates with them and surrender to it. Then, that Eldritch Entity will take care of everything.

When I was young, several people literally tried to make their religion appealing to me with a SURRENDER argument. The logic was:

• The world is not fully understandable by humans. Science does not have all the answers.

• We are at the mercy of many circumstances beyond our control.

• If we accept that G̷͔͊O̵̡̊D̶̟̈́̾ is in control, knows what’s best, and SURRENDER, then everything will be much more bearable.

Even as a child, this logic felt deeply wrong.

My mind is my temple. It is sacred, and I will not defile it with lies of convenience.

And yet, here they were, with a cavalier attitude, casually talking about deciding their metaphysics on what felt good in the moment.

—

Enough on this though, let’s go back to how SURRENDER manifests.

For the reasons above, the quality of this section will be much lower than the one above. I spent much more time with FIGHT people than SURRENDER people and I naturally feel more inclination towards FIGHT.

So, expect my ability to pass an Ideological Turing Test, to comprehend someone who SURRENDERs, to be worse here.

The Naturalist Answer: SURRENDER to 𝒢𝒜𝐼𝒜

There are many different naturalists. The most common ones nowadays are The Greens. But I have met many non-Greens naturalists: New Age people who feel Energy, as well as more rural farmer types.

The common thread is a belief that ＮＡＴＵＲＥ knows best. Quite a few seriously call it ＭＯＴＨＥＲ ＥＡＲＴＨ, and I have heard ＧＡＩＡ in real life.

A large chunk of their belief systems is basically an appeal to nature. Nature is good, nature is beautiful, nature should be preserved, etc.

I think to SURRENDER to 𝒢𝒜𝐼𝒜 is especially naive.

𝒢𝒜𝐼𝒜 was the original Eldritch Deity that we had to grapple with! Our ancestors did their best to survive 𝒢𝒜𝐼𝒜’s constant assaults.

The only reason people can afford to romanticise it is that in the whole Eldritch Pantheon, 𝒢𝒜𝐼𝒜 is the closest one that we got to defeating.

While it might still have control over the oceans, the depths of the Earth, and some particularly inhospitable deserts, we’ve conquered basically almost all of the land territories.

—

While I am sure that we could benefit from listening to 𝒢𝒜𝐼𝒜 in some ways, my immediate reaction is “How about no?”

On the altar of 𝒢𝒜𝐼𝒜, we sacrificed GMOs, nuclear energy, and now artificial meat.

Even on climate change, where one might expect people who defer to 𝒢𝒜𝐼𝒜 to have something to contribute, blind faith makes them counterproductive.

For an emblematic example: Greenpeace and +600 groups, in a letter addressed to Congress, wrote in January 2019:

Further, we will vigorously oppose any legislation that: […] (3) promotes corporate schemes [like] market-based mechanisms and technology options such as carbon and emissions trading and offsets, carbon capture and storage, nuclear power, waste-to-energy and biomass energy.

In general, I tend to think that we need less 𝒢𝒜𝐼𝒜 in our collective thinking, not more.

—

Even worse, many of 𝒢𝒜𝐼𝒜’s servants are explicitly anti-human.

I have heard people celebrate 𝒢𝒜𝐼𝒜’s “revenge” when speaking of climate change. I have met a few people who believe that we are past our “carrying capacity”, that we should have fewer people, that people have too many children, and that this is part of why degrowth is important.

I have heard people celebrate useless regulations because it bothers people and forces them to be aware of 𝒢𝒜𝐼𝒜.

They want to block any solution that would let people live their lives conveniently. The point of forcing them to not use AC, to use unpleasant straws, to recycle, is specifically to be inconvenient enough to count as a sacrifice that makes the issue more salient.

This is in large part why they want to block technological solutions, market-based mechanisms, and enforce degrowth. The latter is the only one that signals subservience to 𝒢𝒜𝐼𝒜 and that we’re humbling ourselves.

—

It’s hard to express how much I despise this way of thinking.

When I’m compassionate, I see FIGHT as misdirected energy. Humans who want to rebel against the Eldritch and take back control, but who don’t know how.

When I’m mean, I see FIGHT as a self-righteous outlet for people who want to hate others and destroy things. And I get it. Like most people, I also feel some violent impulses. I just practise combat sports and play Counter Strike instead.

When I’m compassionate, I see SURRENDER as the hope for a legible Good that one may pray to. Humans who want peace, to not have to constantly fight and finally rest.

But when I’m mean, I see SURRENDER as much worse than a self-righteous FIGHT. I see it as perverse. It is finding joy in replacing humanity with an Eldritch Deity that doesn’t care for us.

SURRENDER TO 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿

There is a vision of the world in which there’s basically only one thing that made the world better.

Not establishing a prosocial culture that values freedom, work, equality and civilisation.

Not the whole humanism thing during the age of the enlightenment.

Not building constitutional republics.

Not the great strides in democratic socialism, providing security and education for all, as opposed to only the rich.

Nope. In this view, the only thing that made and makes things better was 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿. We must SURRENDER to it.

To the extent things improved, it was because we got ECONOMIC AND TECHNOLOGICAL GROWTH!

If we became nicer, it’s because we became richer! The richer we are, the nicer we are. The more economic growth, the better everything gets. Graph goes up means everything more gooder.

The natural conclusion is that to the extent that we want things to get better, we should just accelerate technical progress, and do more capitalism.

If there is a big fertility crash? We should just do some fertility tech and fertility markets.

If some people getting fat lack the discipline needed to eat less? We should do some weight markets and create a discipline drug.3

If there is a big AI problem? With the top AI experts warning of literal extinction risks of humanity? We should absolutely avoid policy solutions, and instead work on safety tech, and safety markets.

If people are too isolated, and have fewer friends, marriage and sex? We should build AI companions and markets for relationships.

—

There is a common bait-and-switch argument among 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 worshippers.

On one hand, there is the quite non-objectionable “All else equal, it is better to be richer.”

On the other hand, we get to the revisionist “Wealth has been the main driver of freedom and social progress.”

I have met several who told me that historically, progress came from economic growth, and thus we should just work to get economic growth and things will improve by themselves.

When I mention other historical factors, they have then claimed that it’s too hard to analyse them. In other words, their analysis is purposefully one dimensional.

They purposefully reject thinking about novel governance, specific regulations or constraints on technology, instead retreating to a vague “growth is good, any constraint on it is degrowth”.

—

In the process of helping with our material problems, 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 has created many others, that are obvious to any human.

People feel extremely alienated, with many perceiving their work as bullshit.

We have the densest cities ever in the history of the world, yet there is almost no community, with fewer and fewer people knowing their neighbours.

People have much less sex, marriages, friends and children.

People are blasted with the lifestyles of people much wealthier than them all the time. At the same time, they are hypocritically told that of course, everyone is equal, and that 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 is universalist.

That any difference is just based on the actual contributions of individuals to society. Never mind that this reasoning does not hold with trust fund and nepo babies, and that 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 simultaneously opposes strongly any type of inheritance tax.

People see that many 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 apostles massively lobby and subvert governments, despite their stated libertarianism.

People see that 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 is trying to automate them, annihilating their only form of bargaining power in a world ruled by 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿.

At the same time, they are told that if they are replaced, it will be with their own good in mind. :)

—

The same way we don’t need more 𝒢𝒜𝐼𝒜, more 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 is not the solution to the problems it introduces.

Even on technologies like AI, that are so risky that it may cause the extinction of humanity from the development of superintelligence, we see extreme amounts of pearl clutching, and a strong resistance from 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 to any type of control.

I have mentioned the ridiculous letter to Congress from 𝒢𝒜𝐼𝒜’s proponents. But the efforts of 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 are no less ridiculous. It tried to pass a federal US bill that would have prevented states from regulating AI for 10 years! Following this failure, the efforts are now continuing in the form of a super PAC to the tune of 200 million dollars.

—

Beyond the mere mismanagement of the risks of extinction from AI, I have witnessed extreme amounts of anti-humanism from 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 apostles.

E/accs are the proverbial butt of the joke here, with their focus on racing to AGI with as little care as possible, and their goal to “maximise entropy”.

But 𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 has tentacles far beyond e/accs, especially in SF.

I once had a conversation with the CEO of a SF-based nonprofit, wherein he told me point blank…4

• People are egoists for caring about their happiness and their children.

• They should care about the trillions of future beings that AGI will enable instead.

• Fortunately, people largely have no power and do not matter, and the only people who matter are the ones building technology.

—

𝒯𝐸𝒞𝐻𝒩𝒪-𝒞𝒜𝒫𝐼𝒯𝒜𝐿 and 𝒢𝒜𝐼𝒜 are both horrific Eldritch entities working against the interest of humans.

But the former is currently winning, and many of its apostles are so captured that they are not even realising what is happening. So in practice, it’s the one that I interact with the most.

Nevertheless, the next one may actually be objectively the worst one.

SURRENDER to The 🅿🅾🅻🅸🆃🅸🅲🅸🅰🅽🆂

This section is about Trump, but largely isn’t.

It is foremost about the childish hope that we can all rally behind a Strong Man, a True Adult, to face the Eldritch in our stead. Or a group of True Adults, like a Political Party.

This hope is doomed to failure in practice.

While it is possible in theory to look at the track record of a politician, check all of the policies that they have supported in the past, see how they helped people, and build an accurate model of them, almost no one does so.

People lack the time, the interest, the intellectual ability, or the background knowledge needed to do so.

So in practice, people follow 🅿🅾🅻🅸🆃🅸🅲🅸🅰🅽🆂 that they know next to nothing about. They know less about their 🅿🅾🅻🅸🆃🅸🅲🅸🅰🅽🆂 than they know about a friend’s friend, and yet they will trust 🅿🅾🅻🅸🆃🅸🅲🅸🅰🅽🆂 more.

—

For instance, of those who follow Trump…

Who knows him?

Who has talked to him?

Who has spent time any non-trivial amount of time?

Who would have expected his policies?

Statistically, almost no one.

In practice, they have no idea who Trump is as a human being. They are not following Trump The Human, they are following 🆃🆁🆄🅼🅿. An Eldritch Deity that has the face of a man, and promises to take care of everything should one SURRENDER to him.

—

The same is true Kamala or Biden’s past supporters.

Despite knowing very little about their political career, their personal or opinion and whatnot, they put all of their loyalty behind these politicians.

Very often, people’s loyalty to 🆃🆁🆄🅼🅿 or 🅱🅸🅳🅴🅽 was enough to sow discord and drive people to insult each other.

For all that, the Eldritch doesn’t care for us, and conveniently uses and abuses us when it stands to profit from it.

The reason why I single out 🆃🆁🆄🅼🅿 is that it is an extreme version of this phenomenon, rarely found in the west.

I think in the last 20 years, in the west, Trump is literally the only major politician who has stated anything close to his people being so loyal, that he could stand in the middle of Fifth Avenue, shoot someone, and not lose any voters.

Conclusion

I think it is quite obvious that neither FIGHT nor SURRENDER are pragmatic answers to the Eldritch question.

They only work at an emotional level. At the intellectual level, they do not create the tools for how Humanity to steer its future. FIGHT is purely oppositional, while SURRENDER is only about sacrificing everything to a chosen Eldritch Deity in the hope that it takes care of things for us.

To be clear, I am not saying that the underlying political movements have zero intellectual content or that they hold no grain of truth.

I am just saying that specifically, they do not pragmatically deal with the Eldritch.

—

I will likely write about my personal approach to the problem later, in a part 3, but this article is already quite long!

On this, cheers :)

The Redistribution Shift

Let’s say that as a matter of policy, we take money from a [contributing group] and give it to a [recipient group].

Then, there will be a knock-on effect: the [contributing group] will tend to shrink and the [recipient group] will tend to expand.

This is The Redistribution Shift. It can largely be understood in terms of induced demand and tax deadweight loss.

The Redistribution Shift can be good or bad. It can be overcome or embraced. It all depends on the situation.

—

If we take money from the active population and give it to retirees, then the active population will be pushed to shrink and there will be more retirees.

If we take money from employed people, and give it to unemployed people, there will be more unemployment.

If we take money from people measuredly abled, and give it to people measuredly disabled, the measured abled population will shrink and the measured disabled population will expand.

If we take money from the middle class, and give it to poor people, the middle class will face a shrinking pressure and the poor population will expand.

If we take money from successful companies, and give it to failing companies, companies will be less incentivised to avoid failing.

If we take money from native citizens, and use it to subsidise immigrants, there will naturally be more immigration.

If we take money from rich people, and give it to everyone else, rich people will get less and less money to their private name in the jurisdiction that is taxing them.

How does it work?

The Redistribution Shift manifests itself in many ways. Some are good, some are bad.

The Margin

Usually, it works by changing the behaviour of people at the margin.

What this means is that whilst there are many people who are firmly contributors or recipients, there are also many who are at the edge.

People who are hesitating about moving out of a country, old people who are hesitating about working one more year, unemployed people who are hesitating about staying unemployed for longer before taking a better job offer.

By taking money out of people, or giving it out to others, the balance tilts for many people. And this keeps happening for as long as the redistribution is on.

Examples of the Margin

The Redistribution Shift works by making more people identify as recipients. Most categories form a spectrum, like being disabled. The more money is taken from abled people and given to disabled people, the more people at the edge will decide to categorise themselves as disabled, and will present to their doctors as such.

This is pernicious. Whilst this makes the lives of recipients better, this constantly incentivises people to see and present themselves as recipient. I have written about some cultural ramifications of this problem in “The Responsibility of the Weak”.

—

The Redistribution Shift works by making it more bearable to be part of a recipient group. The more money someone gets when they are unemployed, the more bearable it is to be unemployed. And the more people at the edge will decide to stay unemployed. A similar effect is at play with retirement, where more people will decide to retire earlier the more beneficial it is.

This is good, but unsustainable. Recipients are usually recipients for a reason. Making their life more bearable is good. But whilst this works well for short-term emergencies, growing the share of the population who takes more than they give doesn’t work: money always comes from someone.

—

The Redistribution Shift works by making the contributors leave. People can be rich enough to easily move out and get, if not citizenship, residency in other countries. Many of them have a breaking point, past which they will decide to stop paying and leave.

This is unsustainable. It is ok to quash some contributions or push contributors out: sin taxes are a useful governance tool. But constantly reducing the size of the contributing population doesn’t work: money must come from someone.

—

The Redistribution Shift works through fraud. The more money is taken from healthy people and given to sick people, the more people will fake being sick. The same is true for veteran benefits, or any type of benefits in general.

This is quite bad. Fraud should not be encouraged.

Direct Impact

Sometimes, the Redistribution Shift works in very direct ways. Rather than by impacting people at the margin, it directly impacts contributors or recipients in bulk.

Here are a few examples.

—

The Redistribution Shift works by preventing deaths. For instance, the Redistribution Shift increases the number of disabled and homeless people by preventing their deaths.

This is very good. People must survive and thrive, and if redistribution can help with it, it’s good.

—

The Redistribution Shift works by changing demographics. Child benefits are specifically about helping poor people with having children, taking money from richer contributors. Immigration benefits are specifically about attracting (when it’s economically useful) or hosting (when it’s socially good) foreign people.

This can quickly become dangerous. Sometimes, people start intending to change demographics through redistribution. It is an admitted goal of many communists that they want redistribution high enough to dismantle rich people as a class. It is an admitted goal in many ethnic conflicts to “redistribute” money from one ethnicity to another in order to pressure them.

What should we do?

When I talk about the Redistribution Shift with people, they often ask me if it means that we should stop all redistribution policies.

This sounds to me like a very extreme reaction to merely pointing out one unintended consequence of a very natural type of policy.

—

In general, political discourse is extremely immature. Arguments are treated as soldiers.

Any case that a policy has a cost or a benefit is taken as being for or against it.

This makes sense when this is applied to politicians (or any influencer) spreading their message to audiences in the millions. They rarely care about the technical merits of their case.

But in actual political debates, treating arguments as soldiers is utterly stupid.

All policies have pros and cons, and we must be aware of them to make the correct choices.

—

So, what should we do?

We should be aware of the knock-on effects of any policy. Sometimes, it is worth paying them, sometimes, it is not. Sometimes, they can even cleverly and cheaply be neutralised or bypassed.

However, when we ignore them, we can easily underestimate their cost or get screwed over by some easily foreseeable problem.

In the context of redistributive policies, here are a few principles that I think are worth keeping in mind.

Redistributive policies work against themselves. After a redistributive policy is passed, people change their behaviour in ways that will accentuate the need for redistribution and make it less sustainable. Thus, it is important to design them well and take into account second-order effects.

I am sceptical of “incentives” in general. I wrote an entire essay entitled “Incentives is Astrology for nerds”.

But national economic policy is one of the few cases where incentives do matter, because people will react, at the very least at the margin. (This is a variant of the Lucas critique.)

Redistributive policies are clearly good when they’re about de-risking. De-risking policies basically amount to strongly regulated insurance programs. For instance, this involves helping with sickness, with accidents, or with punctual unemployment.

In all of these cases, the redistribution policy is net positive. There is a clear demand for insurance there, from almost all of the productive population, and it can make sense to use the power of the state to benefit from economies of scale at the national scale whilst ensuring that nothing nefarious is happening.

Redistributive policies require a lot of trust. Trust in people not to fraud, trust that recipients are grateful, and trust in the government to catch fraud. Absent these, resentment towards the recipients will accumulate, and contributors will explicitly try to avoid contributing.

The level of trust that is required is very high. It is having contributors answer nicely to “If a random person from your polity came and asked you for money because they were retired / unemployed / sick, how much would it take for you to trust them and give them money?”

We do much too little to ensure this level of trust.

Redistributive policies require some give-and-take. To avoid resentment, beyond trust, it is important for contributors to get something out of giving. Otherwise, they’ll try to get out of it and their population will shrink.

This doesn’t need to be much. We humans can be motivated by mere recognition and gratefulness. Most of us work hard for our money, and giving it to others is commendable. So getting thanked for the giving often goes a long way.

We do much too little there too. We very often do the opposite, motivating redistribution not based on kindness and solidarity, but guilt and coercion.

Trying to directly change demographics is a big no-no. Possibly, a more enlightened society might be able to not screw it up. It might carefully investigate the pros and cons, the second-order effects, the impacts on the culture and the civility norms, etc. But no society is there.

In the meantime, I am quite wary of policies like mass immigration, or the renewed interest in technology-based eugenics. They are extremely hard to reverse with far-reaching consequences, far beyond our collective ability to form informed consent over.

Conclusion

Redistribution is a touchy topic.

But it is a very important one. Many people now understand the Government primarily as a Great Redistribution Machine, whose job is to decide who gets the money from whom.

And to some extent, they are right. I rarely see advocacy for Healthcare, Social Security and Public Education as great investment projects, but instead only as redistribution programmes motivated on grounds of fairness and equality.

These spending categories represent an extremely large share of Governments’ budgets. In 2022, it was roughly 50% of it in the US (at the federal level) and in the UK.

This is why it is important to think very clearly about redistribution.

—

Here, I only mention one dynamic among many. This is not a condemnation of redistribution as a whole.

However, I believe it is a very interesting dynamic. I view the lack of foresight around redistribution as a potent example of a much deeper intellectual failure.

The failure to internalise that others react to what we do.

I’ll write more about it later :)

On this, cheers, and have a nice day!

—

In the 21st century, we all live in a cosmically horrific world of our own creation.

Very little makes sense. As we start to understand things and adapt to the rules, they change again.

We live much closer together than we ever did historically. Yet we know our neighbours much less.

We have witnessed the birth of a truly global culture. A culture that fits no one. A culture that was built by Social Media’s algorithms, much more than by people. Let alone individuals, like you or me.

We have more knowledge, more science, more technology, and somehow, our governments are more stuck. No one is seriously considering a new Bill of Rights for the 21st century, or a new Declaration of the Rights of Man and the Citizen.

—

Cosmic Horror as a genre largely depicts how this all feels from the inside. As ordinary people, we are powerless in the face of forces beyond our understanding. Cosmic Horror also commonly features the idea that bad things can happen to us without us noticing, and the angst that comes from it.

The genre often features Eldritch creatures. These are creatures that live beyond humans’ understanding. But whilst humans can’t understand or influence them, they can affect humans. Sometimes, they may even forcefully alter people’s thoughts and actions.

—

As humans, we have always been confronted with the Eldritch. The world was never our home, it was never designed to be legible or pleasant to us.

We come to life screaming and crying, without the intellectual means to understand why we got ejected from a nice place into a deeply inhospitable, cold and uncaring universe.

Many stories echo this. From Hesiod’s Golden Age to the Garden of Eden, we started in a state of primordial ease, where humans lived in a world that was good for them. But something happened, and we landed on Earth, with all of its suffering and hardships.

Even our lives start that way. We all have been children.

As children, we had very little agency. Our fates were in the hands of adults, entities much more powerful than we are. Although adults were uncannily similar to us, they were notably different. They didn’t think the same way we did, we had to play by their rules (or else), and they decided everything around us.

This is how humanity’s history started.

We lived in hostile lands, always at the mercy of death and the elements, without the science and the tools that would let us understand what was happening.

—

Nowadays, our Eldritch is of a different kind.

We thoroughly understand the natural world, but we have deserted it because it sucked.

We have built and migrated to an artificial world.

And yet, it is still full of phenomena we can’t understand, of inhumane entities that decide our fates.

This is existentially dreadful and awful.

There is no myth that makes sense of it.

There is no one to fault.

No one wants this.

Everyone is lost, unable to find a home.

Modern Magic

Most people are utterly confused all the time. They get through by not expecting to understand much.

“Sufficiently advanced technology is indistinguishable from magic.”

Well, aside from a very small STEM elite, technology already is indistinguishable from magic for most people.

Most people do NOT have the expectation that if they study a technical topic, they will eventually understand how it works at a mechanistic level.

They do not expect that they could learn about the state of the art of any field, and eventually make their own opinion.

They just nod, shrug and move along.

They deeply expect that even with more time, they will not understand how phones work, or how cars are built. It is too technical. Possibly, it has something to do with electricity, robots, and circuit boards?

Who knows how a computer, a factory, 5G, or vaccines work? Who expects they could know?

Very few people.

—

Beyond deep hard tech, this also applies to soft technologies. Soft technologies include the Law, governance, how our institutions work, complex processes like containerisation, etc.

For instance, most people treat markets like magic.

Communists ascribe evil properties to markets.

Libertarians treat them as magical tools that can solve all coordination problems.

Nerds largely use them as a divination device: they speak of metaphorical markets, and explain everything through incentives and the efficient market hypothesis.

Aside from a select few people deeply interested in economics and governance, it is hard to see grounded discussions of when markets are practical solutions to practical problems, and how to deploy them safely and productively.

For most people, markets are just magic. Whether they are perceived as good magic or dark magic doesn’t matter, they are still magic.

—

Most people have no hope of understanding complex topics.

Thus, for them, experts are not people to whom we conveniently defer in order to save time. It’s not like they think they could read the state of the art of a field (most have never heard “SOTA”!) and build their own justified opinion.

Instead, to most people, experts are priests, necessarily intermediaries to interpret the scientific scriptures.

Priesthood comes with a lot of authority, and thus necessitates a lot of trust.

Historically, cultures put a lot of importance on acknowledging the special status of priests. The knowledge of a priest is not fungible, not everyone can obtain it. And great power implies great responsibilities.

Priests were de facto privileged. To balance that, their lifestyles usually came with many restrictions, all anchored on discipline and rejecting excesses. Depending on the culture, it might have been forced poverty, forced celibacy, the obligation to follow many additional rituals, etc.

—

On grounds of ideological equality, we have rejected that some people can intellectually understand things that others cannot. Variations in intelligence is a very taboo topic, rarely discussed in grounded ways.

Thus, modern priests are never confronted with their additional moral duties. They are never forced to abide by higher standards. And people resent them.

So much so that following the perceived abuses of the priest class, there has been a recent attempt at a Modern Reformation, mediated by the Internet rather than the Printing Press. Thanks to the Internet, people may not need priests anymore, being free to do their own homework.

The hope was that, through a combination of pop-science, infotainment and gamification, everyone would have become a super polymathic genius! Everyone an expert in hard sciences, soft sciences, the Law, and more!

Instead, we’ve gotten internet wokeism, antivaxx, and more conspiracy theories than we ever needed.

In short, our modern global culture explicitly rejects priesthood, even though there are many fields of knowledge that people de facto do not have access to.

—

Speaking of, no one understands 𝒞𝓊𝓁𝓉𝓊𝓇𝑒.

I outlined one way our culture is changing: the annihilation of “experts” as a class of priests.

It feels real to me, but I might be wrong.

And if I’m right, I’m only right after the fact, I did not predict it.

In other words, I largely do not understand C̲u̲l̲t̲u̲r̲e̲.

I am constantly getting screwed over by my limited understanding of culture. I regularly get surprised by some novel cultural phenomenon that I did not expect, and have to change my course of action accordingly.

This is true of all of us. We are all puzzled and surprised.

C͎u͎l͎t͎u͎r͎e͎ changes much faster than it did 10 years, 20 years, 50 years, 100 years, 200 years or 500 years ago.

Sometimes it’s for the better, sometimes it’s for the worse. But the common thread is that we don’t expect it, nor do we understand it.

Despite its critical importance to our lives, ░C░u░l░t░u░r░e░ is as arbitrary as magic, part of the Eldritch.

Powerlessness

Whilst epistemic horror, the horror of not understanding what happens around us, is a literary genre, I don’t care much for it.

I am naturally a curious person, yet I have made my peace with not knowing things.

Pragmatically though, not knowing things makes us powerless. And this is where true horror comes from.

—

We are all disconnected from everything that matters.

No one expects that they can improve social media, governments, institutions, culture, housing prices, criminality rates around them, etc.

For example, in London, one of the richest cities in the world, there’s a massive wave of thievery that everyone has failed to stop.

Many petty criminals roam the streets looking for phones to snatch. In the last year, 2 fuckers tried to snatch my phone. Shoplifting has exploded.

Lawrence Newport from LFG left a bike in front of Scotland Yard, and it got quickly stolen. Despite having a GPS signal, the police didn’t even check the CCTV nor tried to pursue the criminal. A response sergeant took the time to respond and explain that the police is underfunded, and doesn’t have bike thefts as a priority compared to Domestic Violence and Abuse.

One of the richest cities in the world, in front of its own police’s headquarters, cannot prevent thievery.

Of course, vigilantism and taking matters into our own hands would be punished. And I can see why to some extent.

However, on the other hand, the police is not solving the issue, nor is the mayor, nor is the government, nor are the citizens. And no one is happy about this.

No one understands this state of affairs thoroughly enough to actually change it. Everyone is stuck.

—

It is possible to get unstuck. But the work needed to get there is not easy.

It requires studying a complex bureaucracy, from the funding of the police and diagnosing why courts having a massive backlog of cases, to designing laws mandating stronger sentences where it should.

It then requires making this common knowledge, creating media attention, and putting enough political pressure to finally pass the laws that make sense.

In practice, no one knows how to do this reliably, and people who have tried to change have failed.

Everyone now knows that in practice, thievery is to be expected in London.

C’est la vie. It is what it is. Shikata ga nai.

Thievery is now one of the forces of nature that a puny human has to contend with in London.

In Japan, there are earthquakes. In London, there is thievery.

¯\_(ツ)_/¯

—

London’s thievery is just one minor example of how people feel powerless about a non-natural phenomenon. In the Eldritch Pantheon, it would be a minor spirit.

But there are Eldritch Deities that rule much bigger aspects of our lives.

For instance, Ｔｈｅ Ｅｃｏｎｏｍｙ.

What is 𝕋𝕙𝕖 𝔼𝕔𝕠𝕟𝕠𝕞𝕪? 🅃🄷🄴 🄴🄲🄾🄽🄾🄼🅈 is an Eldritch entity.

Sometimes, Tԋҽ Eƈσɳσɱყ is good. Getting a good job and a home are easy, raising a child is cheap, we can buy a lot of stuff, and everyone is happy.

Sometimes, T̵̤̀͊h̶̲̘̍e̶̻̔ ̴̧̑͘E̶̅̐͜͜c̷̢̥͐͠o̷̢̱̓n̶̬̹͑o̴̠͠m̴̥̝͗̇y̸̢̥͑ is bad. Everything is hard, and we struggle. We struggle to find a job, to borrow money for a home, to educate our children at a cheap and competent school. Everything is expensive, and everyone is unhappy.

If we pray hard enough, the next politician we vote for might ᑕᕼᗩᑎGE TᕼE EᑕOᑎOᗰY.

lol.

ㄒ卄乇 乇匚ㄖ几ㄖ爪ㄚ is not a real thing.

An eCOnoMIsT might say that it is just the combination of 𝔊𝔇𝔓, 𝔘𝔫𝔢𝔪𝔭𝔩𝔬𝔶𝔪𝔢𝔫𝔱 ℜ𝔞𝔱𝔢𝔰, ℑ𝔫𝔱𝔢𝔯𝔢𝔰𝔱 ℜ𝔞𝔱𝔢𝔰 𝔞𝔫𝔡 𝔪𝔞𝔫𝔶 𝔒𝔱𝔥𝔢𝔯 ℑ𝔫𝔡𝔦𝔠𝔞𝔱𝔬𝔯𝔰.

But this is obviously fake.

Everyone would talk about it even without these indicators.

Everyone knows it would still direct our lives.

—

Many other Eldritch Deities direct our lives.

They often look the same.

For instance, I asked ChatGPT to define “Շђє єς๏ภ๏๓ץ”, and it gave me a very Eldritch answer:

“▉▉▉ ▉▉▉▉▉▉▉” is a broad term that refers to the system by which a society organizes […]. It’s essentially the network of activities and relationships that determine […].

This is what the Eldritch Deities look like in 2025. We can’t pinpoint them. We use broad terms. They’re systems related to society. They’re network of activities and relationships.

I know them. You know them. Every modern human knows them.

Every modern human knows they are powerless in front of them.

Every modern human knows they must kowtow to them, if they don’t want to get screwed.

𝕋𝕙𝕖 𝔾𝕠𝕧𝕖𝕣𝕟𝕞𝕖𝕟𝕥, 𝔸𝕔𝕒𝕕𝕖𝕞𝕚𝕒, 𝔾𝕖𝕠𝕡𝕠𝕝𝕚𝕥𝕚𝕔𝕤, 𝔹𝕦𝕣𝕖𝕒𝕦𝕔𝕣𝕒𝕔𝕚𝕖𝕤, 𝕋𝕙𝕖 ℂ𝕦𝕝𝕥𝕦𝕣𝕖, 𝕋𝕙𝕖 𝔸𝕝𝕘𝕠𝕣𝕚𝕥𝕙𝕞, 𝕋𝕖𝕔𝕙𝕟𝕠𝕝𝕠𝕘𝕪, 𝕀𝕕𝕖𝕠𝕝𝕠𝕘𝕚𝕖𝕤.

They’re all systems of networks and activities and relationships of a society.

We may have developed them. They may have arisen organically from our interactions.

But they are here, and we can’t control them. We are ants walking alongside them. They make and unmake our lives. They live by their own rules, amoral and inhumane.

Our artificial Eldritch Deities are here to stay, we are powerless in front of them, and we all feel it.

Escapism and Fantasy

Even though everyone feels it, too many reject the surreality of the situation.

They have not mourned. Instead of believing what is true, they cling to the hope that the world makes sense.

For them, the World should be a Big Story.

The Good Guys succeed, and the Bad Guys should be defeated.

Through a Rightful Struggle, growing and triumphing against our Dark Impulses, we should Defeat Evil and Enact Good.

This is so tragically wrong, naive and pathetic.

—

The typical fantasy of the powerless is that some Evil People have figured it out, and hold The Power.

That we just need to beat them, dispossess them, kill them even, and things will be alright.

This is the pathetic fantasy of the child who hopes there are adults in the room who can make things better when things go to shit.

"The System", "Them", (((They))), The Jews, The Capitalists, The Deep State, The Lizardmen, The Free Masons, The Patriarchy, Israel, The Illuminazis, Politicians.

This is all the same desperate cry.

“Please, have it so that someone has power and could make all of the bad things go away.”

Even if the Evil People do not decide to make all the bad things go away, the fact that they could make them go away would still be reassuring.

At least, someone would know what’s going on.

At least, even if we eventually fail, there would be an Enemy who can make it all meaningful, by Struggling to take them down.

At least, there would be a meaningful story happening, with humans at its centre.

—

But the truth is that no one has power.

Not even politicians and billionaires.

Elon Musk bought Twitter, got Trump elected, and set up DOGE. All that to fail, have Trump pass a bill directly opposed to his principles, and get ousted in less than 6 months.

No one knows how to make things go well. No one knows how, as puny little humans, we could wrestle with Eldritch entities.

This is an unsolved problem, and anyone pretending otherwise is lying.

We suck at building powerful companies that do not end up either corrupting governments, or being throttled by governments.

We suck at having conversations critical to our future without resorting to insults. Picture any conversation about immigration.

We suck at doing redistribution without disincentivising work and without incentivising fraud.

We suck at steering our culture toward any direction of our choosing.

We suck at electing politicians who can reliably pass the laws that Actual Experts my Priests say are good for the majority.

—

This is the core of the 21st century.

Realising that we humans are living alongside entities far more powerful than we are, and that we are losing ground to them.

Gradual Disempowerment has already started, it hasn’t waited for AI.

—

There are more forms of escapism beyond fantasising about a Big Bad Guy.

Giving up and Laying Down and Rotting.

Isolating oneself in the mountains, in a cottage or in video games.

Becoming a selfish asshole.

At best, escapism makes people useless and isolated.

At worst, it makes people panic and hurt each other.

Panicking

When one realises that Eldritch entities are living alongside us, one may panic.

That’s the horror part of cosmic horror. It’s shocking and terrifying.

Unfortunately, panic, horror, shock and terror are not very conducive to wisdom, healthy behaviour, trust and coordination.

—

In their panic, some are looking a bit too hard for a Big Bad Guy. They enter an aggressive frenzy, looking to tear down any enemy near them.

The Woke Left has The Whites, The Males, The Straight, The Coloniser and The Far Right.

The Far Right has Academia, The Swamp, Immigrants and the Woke Left.

The Communist Left has Capitalists, Corporations, The State, and the Fascists.

The Conservative Right has Atheists, Sex Work, Pro-Choice, and the LGBT+.

Because of the biggest propaganda campaign of my time, everyone is also making an opinion on whether Israel or Palestine is a Big Bad Guy.

In general, when people panic, they become more extreme, and fall prey to ideological spirals. This is a major way by which I̷͖̍d̸͈̎e̷̢͑o̸̫̊ľ̸̖ơ̴̠g̶̥̀ȋ̴̫e̷͍͌ș̴͋ grow more powerful.

—

In their panic, a select few try to grab as much power as possible.

On one hand, it might let them take a heroic stance against the Eldritch monsters.
On the other hand… Power is awfully convenient, isn’t it?

This usually doesn’t end well for the people.

Dominic Cummings tried to do it in the UK. He did the Vote Leave campaign and helped Boris Johnson elected. He said that although he thought Boris as a PM was “terrible for the country”, but that “The least bad option seemed to be, exploit the current situation to try and push certain things through and get the country into a better position.”

Thus, he became Boris’ senior advisor, got ousted a little more than a year later, and spent months (or years?) talking trash about Boris publicly, calling him The Trolley, and saying the faster he would go, the better things would be.

Overall, one of the main promises of Brexit (that it would help stop immigration) was broken. Immigration rose to record-high post-Brexit.

A similar thing happened with Trump and Elon.

Elon bought Twitter, used it to boost Trump. He became a “special government employee” when Trump became president, and started DOGE. Elon got ousted 4 months in (much faster than Dominic!). Then he stated that Trump was in the Epstein files and responded “Yes” to someone stating that Trump should be impeached.

Overall, one of Elon’s main promises was to reduce spending deficit through DOGE. In the end, he got ousted around the time of the Big Beautiful Bill, which drastically increased the spending deficit.

—

This panic leads people to playing stupid games, and winning stupid prizes (like Elon or Dominic).

There are very few people who actually try to reclaim power for Team Humanity, in a way that is cooperative rather than by trying to screw over others.

Lawrence Newport’s Looking For Growth is a notable exception. In its short history, it has created a Growth Bill, spread a lot of awareness around it, engaged directly with lawmakers, built a dashboard to clarify what is going on in the UK, etc.

Vitalik’s d/acc is another one. Instead of democratic engagement, it is pursuing the way of technological research, focused on helping people coordinate and enact their will.

The Core Paradox

In many ways, the world is both more and less cosmically horrific than it was 250 years ago.

—

We now understand almost all natural phenomena. Through our global measurement network, we can predict natural disasters before any human witnesses them. Disasters that our ancestors would have pinned on literal divinities.

We now understand biology. When people die, we largely understand what happened, we do not think it was dark magic. We have an understanding of human bodies mechanistic enough that we can design cures, and that doctors can recommend meaningful treatments instead of quack medicine.

We now understand, at least to some extent, psychology and psychiatry. We do not believe in demonic possessions anymore. We understand that mood swings exist, and have documented a large variety of hormonal as well as neurological disorders.

So much of the past belief systems was about the Natural Eldritch Monsters. But through the Great Scientific Method, we learnt to not fear them, uncovered their rules, and discovered that behind their crazy appearance, they were largely explainable in terms of physical and chemical principles.

—

And yet.

250 years ago, we built new constitutions in the US and the EU, and created our modern civilisation.

Now, despite all of our complex technology and science (or rather, because of their complexity), we are struggling to not make housing more expensive.

We are now at the point where we are struggling to have children.

People had children when the entire world was constantly at war, when 50% of children died before 10, when everyone was dirt poor, when no one had Amazon or instant comms.

Now, almost everywhere, we are failing to have enough children to merely hit replacement-level fertility rates.

—

There’s something big that has happened. We went to the moon, but now we struggle to have children? What’s up with that.

My high level explanation for this paradox is…

1) We managed to unravel, scientifically study, and to some extent dominate the Natural Eldritch Deities.

We understand astronomy, plate tectonics, the climate, physics, chemistry and biology.

Although there are still things to be learnt, they are not mysterious, magical or Eldritch anymore. They are regular puzzles, like jigsaws, and we are confident we’ll get there eventually.

2) To get there, we conjured the Modern Eldritch Deities, which come with their own problems.

It seems obvious that we cannot meaningfully steer 𝘈𝘤𝘢𝘥𝘦𝘮𝘪𝘢, 𝘛𝘩𝘦 𝘎𝘭𝘰𝘣𝘢𝘭 𝘌𝘤𝘰𝘯𝘰𝘮𝘺, 𝘛𝘩𝘦 𝘎𝘭𝘰𝘣𝘢𝘭 𝘊𝘶𝘭𝘵𝘶𝘳𝘦, 𝘛𝘩𝘦 𝘈𝘭𝘨𝘰𝘳𝘪𝘵𝘩𝘮, etc. That many problems are arising from this failure.

But it seems equally obvious that getting rid of all of them would be disastrous.

So we are stuck in awkward codependent relationships with these entities.

3) Our Modern Eldritch Deities killed the Traditional Eldritch Deities.

We, modern humans, are not the first to conjure Eldritch Deities.

Our ancestors had already conjured many Eldritch Deities to deal with their own problems. Religions, pantheons, superstitions, ritual systems, and more.

We killed all of them.

In theory, we could have made rational decisions about which ones to keep and which ones to kill. Possibly reshaping some of them, to fit our needs best.

This might have meant: observing Chesterton’s Fence, following a conservative precautionary principle when deciding to stop rituals followed by millions, experimenting with different norms in different places, keeping the rituals and vibes but removing the fictitious beliefs, any of this.

But we did not.

As Nietzsche said:

God is dead. God remains dead. And we have killed him. How shall we comfort ourselves, the murderers of all murderers? What was holiest and mightiest of all that the world has yet owned has bled to death under our knives: who will wipe this blood off us? What water is there for us to clean ourselves? What festivals of atonement, what sacred games shall we have to invent? Is not the greatness of this deed too great for us? Must we ourselves not become gods simply to appear worthy of it?

We killed the sacred. Not only did we kill it, we have eradicated it so thoroughly that we don’t really have the words left to talk about the problems it was solving.

Conclusion

Right now, we are stuck.

We have lost the belief in Progress. That each year will be better than the previous one, for humanity.

We have lost the belief in Agency. That we can do something about it.

We see that we are dependent on entities that do not always want what’s best for us. 𝕲𝖔𝖛𝖊𝖗𝖓𝖒𝖊𝖓𝖙𝖘, 𝕿𝖍𝖊 𝕰𝖈𝖔𝖓𝖔𝖒𝖞, 𝕾𝖔𝖈𝖎𝖆𝖑 𝕸𝖊𝖉𝖎𝖆.

—

We can change this.

This requires a lot of work. The work needed is comparable in scope to the Scientific Revolution, the era of the Enlightenment, or the rise of Formalism.

We will need to transcend our superstitious understanding of the Modern Eldritch Deities. We will need to build a mechanistic understanding of politics, governance, morals and collective action.

Let’s get there step by step. And the first step to defeating the enemy is to name it.

If we don’t do this, we’re condemned to getting screw over by it, never understanding what is happening to us.

—

I have seen too many people feel the bad vibe, never putting words on it, and just getting passively demotivated. Or falling into ideological spirals.

I expect this article will be paradoxically motivating to a few. When I talk to people, it usually frees them a little to name the problem, understand that it is real, that it is not about them, and that this problem is one we can study and eventually solve.

On this, cheers!

I think coordination is very important. Whenever we collectively fail at a large-scale endeavour, I usually diagnose some form of coordination problem.

Whenever I meet people who agree with me on the importance of coordination, they sadly tend to focus on topics I find quite irrelevant in the context of solving coordination problems.

Either they are too theoretical, like mechanism design and game theory.

Or too spiritual, like Buddhism and Spiritual Enlightenment.

Or too intractable, like “improving the culture”.

Prosaic Coordination

The coordination problems I have faced in the past are instead very prosaic.

Here are some examples…

People avoid regularly doing boring things. In many groups I have been part of, regular requirements like writing activity reports or attending meetings on time, have been systematically neglected. The few groups that have made these things happen tend to have put a lot of energy into it.

People avoid conflict. People constantly avoid people who contradict them, and contradicting people. Sometimes, this leads to groupthink, where everyone thinks the same. Usually, this simply leads to dysfunction: people still disagree, but just stop engaging with conversations.

People are not reliable. If someone tells you “I’m going to do [a thing]”, they usually won’t. If someone asks you “Can you please do [a thing for me]?”, they usually will not ping you about it if they still expect you to do it, nor will they tell you if it’s not relevant anymore.

—

Nerds have a natural aversion to coordination. This one is specific to nerds, but I am a nerd, work with a lot of nerds, and expect a lot of my readers to be nerds.

But yeah, nerds have a natural aversion to coordination. Coordination requires going against a lot of nerd instincts. Doing things that one does not fully believe in; doing things that one would not have done by oneself; or deferring to consensus instead of one’s own beliefs.

As a result, nerds avoid some common failure modes: nerds are (usually) not political extremists who scream at people online. But they also develop new failure modes! Like making it impossible to agree on courses of action not fully specified ahead of time.

MicroCommit

MicroCommit aims to solve one specific coordination problem.

How can an organisation, at scale, reliably ask its members for help, and know who will reliably consider helping?

This is a very common problem. Organisations constantly want help from their members: boosting some piece of social media content, stating what they think of a policy proposal, participating in a poll, buying a book, etc.

Right now, there is no reliable way for organisations to ask for this help at scale…

Social media is not the place. Organisations have no guarantee that their posts will reach all of their members.

Newsletters are not the place. It’s very hard to distinguish “I have seen the request and plan to not do it” from “I have not even seen the request or did not consider that it applied to me”

—

This is the problem MicroCommit solves.

The solution is to formalise the notion of “a very small commitment”: every week, taking 5-10 minutes to review requests from an organisation, either accepting or rejecting them, but never ignoring them.

This way, organisations can just see the number of people who reliably (let’s say, over the last 2 weeks!) consider their requests.

Because of my experience with preventing extinctions risks from AI, this is where I will be starting.

ControlAI has a lower engagement way of interacting, namely through subscribing to its newsletter, and I hope MicroCommit can get at least 1-10% of the newsletter’s subscriber count.

Conversely, Torchbearer Community requires a much higher commitment than MicroCommit, and I hope MicroCommit can get at least 50-500x of TBC’s member count.

If MicroCommit works well, I hope that over the next few months, other organisations will be interested in using MicroCommit as well. And if enough do (like 5-10 serious organisations), I’ll likely open MicroCommit for them too!1

—

The current alternative to MicroCommit is for organisations to do a full campaign for each specific request.

For example, if an organisation wanted to ask its members to buy a book, it would need to talk about that specific book for many newsletter emails, change their banners on social media, ping everyone on their community servers (like Discord), et cetera.

Scheduling a full campaign every time an organisation wants to secure the attention of its members is way too expensive.

Organisations would tremendously benefit from the ability to cheaply request 5-10 actions from their committed members every week.

My Hope

My hope is that this can make a dent into the race for cheap attention that social media currently favours, and reward organisations for focusing on people who are more active.

Ideally, I hope to see possibilities for from organisations to move to more substantial engagement requests than “Please QT this!” or “Contribute to this Patreon!”

I want organisations to more often schedule events, debates, giving feedback on proposals for actions, and more generally offering opportunities for short-term volunteering.

Right now, each of these requires a lot of investments, and by lowering this cost, I hope it becomes more common.

Conclusion

I hope you like the idea, and join MicroCommit.

If you care about the future of AGI, coordination and the world, and have 10 minutes to spare a week, please sign up today!

(You may also leave feedback in the comments, or on the form.)

Cheers!

But it is so much more than fun tidbits. This is the story of one of Humanity’s biggest triumphs, and I believe there is a lot to relearn from it.

I hope you will appreciate reading it.

Formalism is the view that maths and logic are primarily about syntax, what one can do within very precise rules of symbol manipulation. This is opposed to maths being about numbers, shapes, or a world of abstract ideas.

Formalism underpins our modern understanding of maths, logic and computer science. It is the core insight that led to the computing revolution and our modern understanding of scientific modelling.

Thanks to our now-formal understanding of logic, we built machines that can automatically check mathematical proofs for us. It’s interesting to reflect on how we got there.

—

Alas, I am not a maths historian.

Fortunately, I am a writer! I have a story of maths that I like to tell, and people like to hear it.

Its key points are all factual.

This is by no means the only story of maths one could tell. For instance, one could focus on social aspects, like the rise of scientific education. That story would connect math progress to the Golden Age of Athens, the Islamic Golden Age and the Age of the Enlightenment.

Nevertheless, my story is about formalism, and I don’t think I have ever seen the story of maths spelled out this way.

—

One of my first essays on this website was on Mathematical Formalism. I was a worse writer back then, but it is short and I recommend reading it, as it will help understanding what I am talking about.

So, without further ado, here is my personal take on the history of maths.

(Some parts get quite technical. Feel free to skip them if you can’t follow, they’re not critical. Or ask questions in the comments about them!)

Summary

I divide the History of Maths into 4 eras:

1. The Informal Era (–14th). People did maths that was mostly about numbers, and they did so in prose. It was painful.

2. The Notational Era (14th–19th). Mathematicians discovered and standardised notations. This led to an explosion in maths with many fields being created.

3. The Foundational Era (late 19th–1930s). Mathematicians, puzzled by the new heights they reached, sought to discover the nature of maths.

4. The Formalist Era (1930s–). To a large extent, they succeeded! We now consider their solution obvious, and have built most of our techno-scientific stack on top of it.

This is a major adventure, a big win for team Humanity.

We need more wins like this!

The final section lists a few learnings from this win, that we ought to apply to our own problems.

The Informal Era: Prosaic Arithmetic (until ~14th century)

For most of history, maths was pretty informal.

But what do I mean by informal? Isn’t maths the most formal thing?

What I mean is that it was mostly text and people making arguments through text. There was basically no standard mathematical notation aside from numbers.

For instance, the “+” symbol dates from the 14th century, and the “-” one from the 15th! So, when talking about additions, people just did whatever. Some put numbers next to each other, others would say “_ and _” or “the sum of _ and _”. Everyone had their own conventions!

—

Wikipedia has an early algorithm (written by Al-Khwarizmi, whose name led to the word “algorithm”) used to solve a second-degree polynomial.

This algebraic example, from the analogously eponymous book Al-Jabr and Al-Muqabalah (whose title led to the word “Algebra”), is illegible. To me, it reads like hard-to-understand mumbo-jumbo.

It starts with:

If some one says: "You divide ten into two parts: multiply the one by itself; it will be equal to the other taken eighty-one times."

What does this even mean! (And it only gets more confusing as you read more.)

To be clear, this was not another set of conventions, which could be canonically translated to our modern notation. This is just everyone writing down their thoughts in prose, in their own peculiar and internally inconsistent ways.

—

In practice, during the informal times, maths was either accountants counting things, or geniuses writing down their intuitions in prose. Said intuitions were mostly related to arithmetic, which is about counting things in ways more complex than accountants did.

Because all of this was in prose and intuitive, merely understanding someone’s text required a lot of intellectual horsepower.

Some, like Al-Khwarizmi and his algorithms, understood or felt the necessity of having non-ambiguous procedures to do maths.

Of them, a select few went deeper, and aimed to develop non-ambiguous procedures to do logic. Not merely shuffling numbers around, but ensuring that the reasonings and proofs themselves were correct.

I have two go-to examples of this.

—

1) Euclid, who first developed a proper axiomatisation of planar geometry, in his Elements.

This one is close to my heart. When I was young, I was quite fascinated by the type of Straightedge and Compass constructions that one may find in the Elements.

They made for an infinite source of puzzles: given a target drawing, is it possible to define a procedure that lets one do it with only a ruler and a compass?

It was both interesting from a mathematical standpoint, and resulted in beautiful constructions. Even the simplest rosette patterns (“rosaces” in French) like the one below can be quite satisfying.

—

2) Aristotle, who was one of the first to develop a proper rule-based logical system, in Prior Analytics.

It is impressive, as it is one of the earliest attempts to formalise logical reasoning itself.

But aside from its historical context, it holds little value. Like, consider this example, which was later nicknamed in the Middle Ages “Barbara”.

If A is predicated of all B, and B of all C, then A must be predicated of all C.

This just means “If all humans are mammals, and all mammals are animals, then all humans are animals”.

Nevertheless, Aristotelian syllogisms represent a non-trivial fragment of logic, which was considered relevant for a long time.

(If you’re curious, the fragment is the “Monadic Predicate Calculus”, but it doesn’t really matter.)

—

These exceptions are brilliant in how prescient and modern they are. They got so close to our modern formal proof systems!

So close, yet so far. They were missing pretty big components of what was needed to get there. The biggest of them being Notations.

The Notational Era: Maths’ Cambrian Explosion (14th to 19th century)

Simple Symbols (14th to 17th century)

I have already mentioned that “+” and “-” are only like 500 or 600 years old.

Well, they were so convenient that people loved the idea, and started to use more and more symbols.

In an addendum to his Discours de la Méthode from 1637, Descartes writes La Géométrie. In it, he introduces the now-standard notations of x, y and z for variables; as well as a, b, and c for constants.

—

Le Discours de la Méthode is a great book overall. It is quite short, but in it, Descartes introduces, among other things:

• The Cartesian Doubt: an approach to epistemology based on extreme scepticism. The OG “source?” guy.

• The famous banger “Je pense donc je suis” (“I think, therefore I am”)

• Major contributions to optics

• A few variants of the ontological argument for the existence of God

And more to our point, he introduces Cartesian Coordinate systems, that let people cleanly bridge the intuitions they had from arithmetic and geometry.

It’s not the best book, for instance, Wikipedia says “Descartes justifies his omissions and obscurities with the remark that much was deliberately omitted "in order to give others the pleasure of discovering [it] for themselves."”

(This is the original “The proof is left as an exercise to the reader.”)

In the same glorious year of 1637, Fermat famously writes that he has a proof of his eponymous theorem, but that it was too big to fit in the margin. (The joke is that the theorem was only proven in 1995 by Wiles, using advanced maths that I can’t understand and that Fermat could never have produced.)

—

Still in the 17th century, Descartes’ contemporaries introduce more and more symbols! Multiplication, division, infinity, pi and more.

Building on top of these new notations, Leibniz and Newton independently both introduce their own notations for what became calculus. One of the biggest shit shows in the history of maths ensued where people fought over who plagiarised whom.

—

In general, the 17th century was a great time for maths. People started using compact notations for what would have taken pages of prose, which let them experiment much faster with their ideas.

Notation was also much more visual, making it easier to spot mistakes. Compare Al-Khwarizmi’s algebraic text to this image:

They express the same concepts.

“Al-Jabr and Al-Muqabalah” specifically referred to “The Balancing Method”, it literally meant “To Complete and Balance”.

Yet, the visual aspect of mathematical notation is just far superior to written prose.

—

These benefits of notations were crucial.

On one hand, notations let mathematicians explore faster. To get to the same place, they had to write much less. Thus they could try more things faster.

On the other hand, notations let mathematicians explore more confidently. It is much easier to check one’s errors in visually arranged equations than in dense prose.

Euler (18th century)

And thus arose the King.

Euler has a long Wikipedia page of topics named after him.

He is also famous for this identity, often described as the most beautiful equation in maths.

In this identity, e, i and π are all notations that he introduced or standardised.

(I always thought “e” stood for “Euler”, but actually, Wikipedia says it is unknown why Euler picked “e” 😊)

Basically, he took all the notations of his time, introduced some of his own, and industrialised the notation process.

By that, I mean the following:

1. He combed through all the fields of maths that people before him had explored.

2. He re-analysed each field with a standard set of notations, introducing his own whenever needed.

3. Using this, he grabbed all the low-hanging fruits. He proved all the results that are “easy” (assuming you’re a genius) to prove when you have access to good notations.

And it turns out that “all the low-hanging fruits” was a lot: he became the most prolific contributor in maths history.

Here are some quotes from Wikipedia to show I am not exaggerating:

Pierre-Simon Laplace said, "Read Euler, read Euler, he is the master of us all"

Carl Friedrich Gauss wrote: "The study of Euler's works will remain the best school for the different fields of mathematics, and nothing else can replace it."

In an effort to avoid naming everything after Euler, some discoveries and theorems are attributed to the first person to have proved them after Euler.

Euler marked a turning point in maths.

Until then, maths was still quite practical in one way or another. For example, Newton was interested in calculus in the context of the laws of motion (where speed is the derivative of position), and Leibniz was interested in calculus in the context of calculating the area under a curve.

Such practicality restricted maths. It held mathematicians back with physical interpretations for most of their concepts, as opposed to embracing the nature of maths as a formal science that only cares for symbol manipulations.

But Euler’s work changed this. It demonstrated how powerful a fully formal approach could be. Unconcerned with the concrete interpretation of its terms, simply applying the same abstract proof techniques to many domains, he single-handedly advanced maths a century forward.

Abstraction (19th century)

The 19th century saw the birth of truly abstract maths.

My favourite example of this is all the fuss around Euclid’s 5th Postulate.

In his Elements (which we mentioned earlier), Euclid introduces 5 postulates. They are meant to capture the intrinsic properties of his geometrical setting, which is the 2D plane.

The first four postulates state:

1. Between any two points lies a single straight line.

2. Any line segment can be infinitely extended.

3. For every point and radius, there is one and only one circle associated with it.

4. All angles formed by perpendiculars are equal.

All are simple, and obviously correct.

Like, we would not naturally think of them as working assumptions or axioms.

We would think of them as true facts about the world, like, “Of course between two points there’s only a straight line!”

—

The fifth postulate is a bit harder to explain. So I will use an alternative one that is equivalent (assuming the first 4).

This alternative postulate is called Playfair’s Axiom:

Given a line and a point not on it, [there is] one line parallel to the given line can be drawn through the point.

This fifth postulate stunned many mathematicians.

It felt redundant to them. Given Euclid’s definitions and his first four postulates, they felt that one ought to be able to prove it from the first four.

And if you know some maths folklore, you know the punchline: they were wrong.

It is in fact not possible to prove the fifth postulate given the first four! For instance, in hyperbolic geometries, the first four postulates hold, but not the fifth.

Even worse, the first four postulates which we called “obviously correct” earlier? Welp, on a sphere, the first and third postulates do not even hold!

—

This is representative of the tension at the core of mathematical abstraction.

If maths was about numbers or shapes like lines and circles, then in truth, we don’t even need axioms. We are just talking about natural phenomena that we can observe, similar to the laws of physics.

But if we adopt the formalist thesis, that maths is specifically about building axiomatic systems and seeing where they take us; then new horizons unfold.

Until the 19th century, mathematicians were too stuck in a concrete mindset where geometry had to be about Euclidean spaces. Not because they made a conscious choice to study what we now have formalised as Euclidean Spaces. But because this was prima facie obvious, and our first impressions are not that good.

They did not imagine that one could build new abstract objects by artificially removing axioms that felt true and obvious. Let alone imagine that these new abstract objects would be interesting in themselves, and that non-Euclidean spaces would help us accurately describe our universe!

—

Over the course of the 19th century, the mindset of mathematicians changed.

People built non-Euclidean geometries for breakfast.

People built abstract algebraic systems. Where Al-Jabr was the act of balancing equations of numbers, these abstract algebras did not involve numbers. It spawned the flora of monoids, groups, rings, fields, and other delights.

People built abstract polynomials, which transcended polynomials over numbers. This spawned algebraic geometry and more.

This is the beginning of modern maths! Super abstract, and hard to immediately intuitively grasp.

For instance, for concepts that predate modern maths, the motivations were roughly understandable. Newton was interested in differentiation because of position, speed and acceleration being related to each other through it.

But for modern things like Weierstrass’ Elliptic Functions, the “motivation” section doesn’t make sense to normal human beings anymore. It is transcendently abstract. Consider:

The Foundational Era: Questioning Everything (end of 19th and start of 20th century)

The Foundational Crisis (end of 19th century)

Even if formalism was not fully defined and established as a philosophy of maths, it became obvious to maths was transforming.

Maths became less concrete and less hypothetico-deductive. The practice of maths became much less about studying ideal ideas (like numbers or squares) and their properties, as Plato once described.

It was turning more abstract, with mathematicians defining new objects, new systems, and looking at what was happening.

Mathematicians were thus wondering…

If maths was not about real things, what was it about?

This question was the source, itself upstream of many more practical questions.

What even is abstraction? Saying “Maths is the study of abstract things” is not satisfying if we don’t really know what abstraction is.

Why was notation so effective? How come that in less than two centuries, there had been much more progress than in the entire rest of human history?

Why are mathematical proofs so much more solid than other types of proofs? Like, if I am persuaded by a mathematical proof, I will be wrong much less often than when I am persuaded by a philosophical proof. Mathematical proofs are just much more solid.

Despite being so solid, why do mathematical proofs sometimes fail? Sometimes, division by zero can work. But usually, it doesn’t. How do we know when it’s going to fail?

—

Everyone tried to come up with their answers, trying to establish proper foundations for maths.

Some did it in their specific sub-domains.

For instance, Cauchy and Weierstrass were both interested in a proper definition for the infinitesimals used in differentiation. This is what led to the modern ε-δ notion of limits.

Others tried to come up with alternative axiomatic foundations for geometry, to answer the shortcomings of Euclid’s.

Still others went for the true definition of numbers. This led to the Peano axioms for the natural numbers (a definition of numbers based on properties of addition and multiplication) and Dedekind cuts for real numbers.

—

But from my point of view, the most interesting endeavour was to try to understand what all of maths was about. What were proofs? What were mathematical entities?

This is the true foundational crisis.

And when people tried to come up with a universal answer like set theory, paradoxes abounded.

Instead, people started moving towards more pragmatic questions. Like, “How could a mathematician, using an effective procedure, precisely solve this pragmatic problem?”

In these questions, the notion of an “effective procedure” was often mentioned. This is the direct parent of our modern-day algorithms.

Hilbert’s Problems (1900)

To deal with this foundational crisis, Hilbert used a great intellectual trick, too often underrated.

He wrote a list.

He took what he thought were the most revealing problems of his time, phrased them in the most useful way he could think of, and listed them.

Where before, the foundational crisis was a bit too philosophical, it became much clearer.

Hilbert simply wrote down all the important problems of his time.

I believe that in his list, these three problems were especially important.

The Second Problem

Proving that the axioms of arithmetic are consistent (ie: that they do not imply a contradiction).

This is a great intuition: if one can prove that the axioms of their system are consistent, then they have established that there is no paradox and that the system in question is solid.

This was partly motivated by all the paradoxes and inconsistencies that came from early versions of set theory.

The Tenth Problem

Building an effective decision procedure to check whether a given polynomial Diophantine equation with integer coefficients has an integer solution.

This is yet another great intuition: the goal is to build a mechanical decision procedure. On Wikipedia, it is described as him asking for an algorithm; but this is anachronistic. This problem helped consolidate the intuition that would later become the algorithm.

The Twenty-Fourth Problem

Developing a general theory of the method of proof in maths.

In my opinion, this problem is the most important one, and unfortunately went unpublished!

This is him calling in advance what will become the study of formal proof systems, and his future programme.

This problem goes very deep.

For instance, Hilbert considered that an answer to this problem should let us formalise the notion of which proof of a theorem is the simplest.

This “simplest proof” concept was much more refined than even our modern study of proof length complexity.

Hilbert’s Programme (1920s)

Thanks to Hilbert’s list, a lot of progress was made toward formalising proofs, logic, consistency and other similar concepts.

The culmination of these efforts was Hilbert’s Programme.

The idea of the programme was to ground all of maths in a minimal axiomatic formal proof system.

The system ought to be definitively proven coherent (formally free of contradictions) to avoid the past misfortunes and paradoxes.

Once this was done, the final goal of the programme would be to define an effective procedure that could take any statement expressed in this formal system, and prove whether it is true or false, still within the formal system.

This challenge was called the Entscheidungsproblem.

Hilbert’s challenge was extremely ambitious! His whole programme was about creating an operating system to then define a procedure that could solve all of maths.

But it might have been too ambitious.

Gödel’s first Incompleteness Theorem (1931)

The first blow to Hilbert’s programme was Gödel’s first incompleteness theorem.

This result has been pseudo-intellectualised so much. It’s comparable in scope to how people use quantum mechanics to “prove” things about consciousness and drugs.

For instance, people try to use it to show that humans can necessarily do more than machines and there will be human ethical truths that machines can not come up with. The incompleteness theorems’ misuses are so pervasive that an actual book has been dedicated in large part to them.

But no, Gödel’s result is relatively simple. And it is purely formal.

Let’s take a less confusing example first.

Gödel’s Law

Most governments can pass pretty advanced laws. Like, it is possible to pass:

• A normal law that says “Blue helmets are forbidden”

• A meta law that says “It is forbidden to pass a law forbidding blue helmets”

• A self-referential law that has a clause that says “This law will expire in 2 years”

Well. If a legislator were so inclined, they could submit the Gödel Law: “This law can never be passed into law.”

If the code of law is sound, then the Gödel Law can never pass, as it would contradict itself.

But then, if you think about it… It means the Gödel Law is already in action! The Gödel Law precisely specifies that it can never pass.

Thus, the code of law is not complete: even though the content of Gödel Law is true, or de facto in effect; it may never be so de jure.

This is a fun quirk. And except if we wanted the code of law to be able to logically codify everything, we shouldn’t be too sad about it.

Specifically, we should not infer from the Gödel Law anything about what the code of law can regulate in practice. This has no bearing on practical matters.

This doesn’t tell us how hard it is to regulate markets, presidential terms or anything that is not deeply self-referential.

The Gödel law doesn’t tell us anything about policy, political philosophy, or governance. It is purely a formal artefact that makes constitutional lawyers laugh.

Gödel’s Proof

Well, now, let’s move on to formal proof systems. Gödel has basically shown two things.

1. If a proof system is powerful enough, it can represent itself.

Gödel has shown that in an Advanced enough Proof System (let’s call it APS), one can write:

• Concrete statements like “2 + 2 = 4”

• Meta statements like “APS admits a proof that 2 + 2 = 4”

• Self-referential statements like “APS admits a proof of this self-referential statement”

Specifically, he has shown that the proof system doesn’t need to be that advanced. It just need to have an encoding of basic arithmetic: natural numbers, addition and multiplication. His method is called Gödel numbering, and it’s quite convoluted.

—

2. A sound proof system that can represent itself has statements it cannot prove.

He has also shown that in particular, he could write a Gödel Statement that says “This very statement is not provable in APS”.

If the proof system is logically sound (it can only prove true things), then the Gödel Statement cannot be proven. Proving it would be an example of proving a false thing.

Thus, it is true that the Gödel Statement is at the very least not provable.

But then, it means that the Gödel Statement is true, as what it says is that it cannot be proven!

So it means that there is a statement that is both true and unprovable in APS.

This is what we mean by “incompleteness”.

That’s it.

It’s not about there being “intrinsically unknowable things” or any of the bullshit interpretations that I have read.

This doesn’t tell us anything about the human spirit not being study-able by science or maths not being able to talk about art.

It’s just a formal thing that is irrelevant to most people and even most mathematicians.

—

But this is enough to defeat Hilbert’s Programme!

One should remember that Hilbert’s Programme was extremely ambitious. It was a universal hope, and any single counterexample thus suffices to shatter it.

Follow-ups (1930s)

After Gödel’s first incompleteness theorem, a few other theorems followed, following the same template, dealing subsequent blows to Hilbert’s programme.

Tarski’s Undefinability Theorem

Tarski’s undefinability theorem (from 1933) says that in a proof system powerful enough, one cannot define a predicate for truth.

The proof is even simpler conceptually than Gödel’s!

Let’s assume that in my proof system, I can say “_ is true” for any sentence.

Let’s further assume that my proof system is powerful enough for self-referentiality, and that it has negation built-in.

Then I can write the Tarski Sentence “This sentence itself is not true”, which is a direct instance of the liar paradox.

This is not about whether we can prove the Tarski Sentence or not. This is about whether it is actually possible to even formally define it in a proof system self-referential enough. And the answer is no.

So, for the very same reason that it is not possible to define an oracle that is always truthful if it is allowed to say “I am lying”, a strong enough formal proof system cannot define a predicate that is always truthful.

This is exactly as profound as the liar paradox.

Gödel’s Second Incompleteness Theorem

Let’s keep going!

Gödel’s second incompleteness theorem is even more self-referential.

Feel free to just jump to the bolded sentences below if you don’t want to hurt your mind with more self-reference.

Assume an Advanced Proof System (APS) is strong enough to be self-referential, then we can express…

• Concrete statements like “2 + 2 = 4”

• Meta statements like “2 + 2 is provable in APS”

• Self-referential statements like the Gödel Statement “I am not provable in APS”

In APS, we have seen that the Gödel Statement was not provable, and yet it was true.

However, it is provable that “If the Gödel Statement is provable in APS, then there is a proof of contradiction in APS”.

By contraposition, we get a proof that “If there is no proof of contradiction in APS, then the Gödel Statement is not provable in APS”, which is just “If there is no proof of contradiction in APS, then the Gödel Statement is true”.

Thus, we prove that there’s no contradiction in APS, else, this would let us prove the Gödel Statement correct, which we know is not provable from his first incompleteness theorem.

This is a much more direct hit to Hilbert’s Program: it means a non-trivial proof system cannot prove that it is free of contradictions.

Turing’s proof

The final nail in the coffin to Hilbert’s Programme was from Alonzo Church and Alan Turing, both independently proving its final goal was impossible in 1936.

I’ll stick to a simplification of Turing’s proof, because it’s the most accessible one, and arguably the most impactful one too.

The proof comes from his paper “On Computable Numbers, with an Application to the Entscheidungsproblem”.

It features several key components…

—

1. Turing starts by defining “automatic machines”, which are now called Turing Machines.

This is arguably the birth of computer science.

Turing defines a minimal formal model of computation, for the purpose of proving Hilbert wrong.

Nowadays, his model of computation is still studied as such, so much so that an algorithm could nowadays be defined as “whatever can be implemented in a Turing Machine”.

—

2. Turing then defines the Halting Problem, the problem of writing a Turing Machine that predicts whether another Turing Machine runs forever or eventually halts.

Turing shows that solving the Halting problem is impossible, using a proof technique very similar to Tarski and Gödel: a diagonal argument.

For simplicity purposes, I’ll explain the proof using the modern Python programming language instead of Turing machines.

Let’s assume that there was a Python program that let us know whether a program halted or looped forever. Let’s call it “oracle”.

Then we could just write a “liar” program that is meant to defeat the oracle. It just looks at what the oracle predict it would do, and then does the opposite. In Python, this is:

If the liar sees that the oracle predicts that it would halt, then it decides to run forever.
Else, if it sees that the oracle predicts that it runs forever, then it halts immediately.

Thus, in Python, as well as with Turing Machines, it is not possible to have a program that solves the Halting Problem.

—

3. Finally, Turing proves that if we could write a program to solve the Entscheidungsproblem, then we could use it to write another one to solve the Halting Problem.

This is proof by reduction, which is about transforming a problem into another one.

And this reduction does something very deep. It shows that the process of finding a proof is general and powerful enough to encode any arbitrary computation.

In logical terms, he shows that we can write the following logical formula, showing that there is some time value for which a Machine halts on some given Input.

Thus, an algorithm to find a proof for an arbitrary statement would let us solve the Halting Problem, which we know to be impossible.

Thus, there is no such algorithm.

This is huge.

This basically turns Hilbert’s Programme on its head!

Hilbert’s Programme was about defining an algorithm to prove everything.

Instead, Turing had shown that it was impossible, precisely because one could use proofs to implement any algorithm!

The result of the Programme

The 1930s were the culmination of more than 50 years of work on the foundations of maths.

Through Hilbert’s list of Problems, his Programme and his Entscheidungsproblem challenge, mathematicians got a precise roadmap to develop a deep understanding of formalism.

Before then, people were lost on how to deal with the nature of maths. It seemed overly philosophical or metaphysical.

But Hilbert’s formalist agenda attempted to mathematically ground it. To definitively answer a metaphysical question through maths.

Then the work of Gödel paved the way for the formal treatment of self-reference.

Then the work of Tarski established a clear separation between formal syntax and intuitive meanings (semantics). The culmination of which would later be Model Theory, the field of formally treating semantics.

Then the joint work of Turing and Church created our modern understanding of computability which is captured in the Church-Turing Thesis, the idea that all formalisms for “effective procedures” are roughly equivalent.

All of these were about the strict application of simple formal rules.

We set out to fully resolve a question which appeared to be philosophical, as vague as the nature of maths. We put our brightest minds to it.

In the spans of a few decades, we created a full theory of mathematical proofs, of formal truth, and of computation.

This may have not met the high expectations of Hilbert, but it’s still got about everything else going for it.

The foundational crisis got resolved. Instead of a single sharp answer, its resolution was more one of progressively building over decades an understanding of formal maths and of its limitations.

The Formalist Era (1940 onwards)

The Answers

So, what was the answer of formalism to all the questions we had?

On the nature of maths

Maths is not a science where one makes hypotheses and tests them. It’s unlike a natural science or a social science.

It is a formal science! This means it is fully abstract.

I explain more about it in my essay on formalism, but quickly: it means that it is about symbolic manipulation, regardless of any meaning we could ascribe to these symbols.

On the effectiveness of notation

Before notations, mathematicians were only relying on their intuitions and platonist preconception of various concepts.

This intrinsically limited them to the strength of their imagination and of their intellect.

With notations, it became much easier for mathematicians to transcend their naive intuitions, by only focusing on the formal rules of maths regardless of what things truly meant.

Instead, with this new way of doing maths, the proofs became the ground truth, and intuition were only a source of inspiration.

This empirically let them go much further than their intuitions, pushing calculus past its physical grounding, pushing geometry past its euclidean interpretation, and so on.

Euler was the biggest example of this.

In essence, by letting people mindlessly apply derivation rules through compact notations, it separated the activity of coming up with interesting formal rules from the activity of coming up with situations where these formal rules directly apply.

—

For instance, when one manipulates the axioms of Peano Arithmetic that defined the numbers in terms of the properties of addition and multiplication, they are not merely dealing with the standard understanding of natural numbers.

There are structures, where the axioms of Peano Arithmetic are valid, that are nevertheless “bigger” than just the natural numbers. These structures are called non-standard models.

Non-standard models are the key to another insight about Gödel’s incompleteness theorem, which is that the truth of the Gödel statement is actually relative to the type of numbers we have in mind when we think of Peano Arithmetic.

Past understandings of maths would have completely ignored such non-standard models, and would not have even realised that the axioms of Peano Arithmetic could not fully capture what we meant by natural numbers.

It is only by following notations without encumbering ourselves with preconceived interpretations that we managed to get there. And the more we went through the motions of notations, the more we got out of them.

On why mathematical proofs are so much more solid than other types of proofs

This comes from the fact that we can actually list all of the axioms and all of the deduction rules that we use in formal proof systems.

This lets anyone replicate any proof, so much so that we can have computers do it. And it is what makes formal proof much more reliable: it is very easy to find mistakes in them.

Furthermore, by virtue of listing all the axioms, we can standardise which axioms we use and collectively benefit from their study.

For instance, we talked a lot about the consistency and completeness of Peano Arithmetic, but a lot of similar work was later done on ZF/ZFC, a foundational set theory more practical than Peano Arithmetic for “everyday” maths.

People did a lot of work around the consistency of ZF and other related axiomatic theories. People stress tested these axioms, checked in which circumstances they were necessary, etc.

This standardisation makes maths especially resilient, even compared to other formal fields, where the axioms used in practice are not necessarily standardised across its practitioners.

On why mathematical proofs nevertheless sometimes fail

The formalist account of maths gives a complete answer to this question!

Namely, either…

1. The formal proof system in which the proof is expressed is inconsistent.

2. The specific formal proof does not follow the rules of the formal proof system.

3. The proof is correct, but is applied to a situation that does not precisely match its axiomatic requirements.

For example…

1. Defining new formalisms creates a risk of (1). If they are not well thought through and battle tested, they risk being proven inconsistent.

2. Doing maths “intuitively” creates a massive risk of (2), where it feels like we have the full proof in our mind, but we actually start feeling stupid when we try to formalise it with a proof assistant like Coq.

3. Any philosophical mishandling of maths is (3). It may be any of the countless misapplications of utilitarianism and game theory, or the eternal spiritual torture of both Gödel’s incompleteness theorems and quantum equations.

Thanks to formalism, we now have a mechanistic understanding of what an invalid proof is! It went from something extremely metaphysical (“Sometimes, we may have arguments that sound correct to people and yet end up wrong”) to an object we can encode in a fully automated machine.

The Applications

Ok, I talked a lot about what formalism meant and sang praises to it.

But the proverbial proof is in the pudding.

So how cool has everything gotten since we started using our formalist understanding everywhere?

The Modern Computer

There were computers before 1930. But almost all of them were special-purpose arithmetic engines.

For instance, the Arithmometer was a desktop computer that let one add/multiply/subtract/divide two numbers. The Arithmometer held a de facto monopoly for 30 years in the entire world, which is hard to imagine nowadays.

Similarly, IBM got started with the Tabulating Machine, which was basically a machine special-purpose built to read and help aggregate information stored on punch cards (the mechanical ancestor of memory cards). This is the oldest system that I would count as an Excel. The fun story here is that a particularly bad US census (which took 8 years to complete) motivated the creation of the machine.

—

There were some who thought beyond special-purpose machines.

The idea of an engine that could process many more types of operations had already been in the air for a century, thanks to Babbage’s Analytical Engine from 1837. This was a prescient design of a machine that supported all the basic algebraic expressions, as well as some conditional and loops.

From Babbage’s design, Ada Lovelace suggested a program that would have let one calculate Bernoulli’s numbers using the Analytical Engine.

But the Analytical Engine was much too complicated: it was meant to work in decimals as opposed to binary, with a 1000 numbers, and built-in functions for all of arithmetic. Thus it was never built. The vague idea of “generality” was not enough by itself.

—

Much later though, as part of his proof rejecting the Entscheidungsproblem, Turing had to define a new type of computer, one that would be self-referential enough to define its own working rules as a program, represented in memory. The new type of computer was later called Turing Machine. And the special program encoding its own rules was the Universal Turing Machine.

It seems obvious nowadays, but back then, programs were separate from memory. The insight of code as data is modern! Without it, it would have been extremely hard to build operating systems, programming languages, or static analysers. And it would have been straight-up impossible to have self-modifying code or dynamic optimisers.

But this insight is precisely what was needed to reject the Entscheidungsproblem! Turing’s proof started by defining a programming model that was powerful enough to be general and self-referential, yet small enough (as opposed to Babbage’s designs) such that it could easily be reduced to first-order logic with Peano Arithmetic.

This new computing model, the Turing Machine (and soon after the equivalent but more practical Von Neumann Architecture) was directly responsible for computing as we now understand it.

Computational Complexity Theory

The 1930s were mostly focused on Computability, which was the question of whether there existed “effective procedures” for various problems.

Here, “effective” is a bit of a misnomer. Back then, “effective procedure” just meant “mechanical and deterministic”.

But now, we live in a world that has formalised computation and has done so for decades. When we think about the efficiency of a procedure, we think about more than “Is it theoretically possible to define a program that performs a specific task”.

We think primarily about its speed, and its memory consumption.

The field dedicated to studying this (among other things) is Computational Complexity Theory, and is directly downstream of the formalist tradition.

Its basic techniques are actually the same as the ones used in computability! Self-referential contradictions, as well as formally reducing problems to others.

Beyond formalising the considerations around the efficiency of procedures, Computational Complexity Theory lets us formalise a large array of real-time trade-offs.

Computational Complexity Theory has many philosophical implications which leads to a more refined vision of the world. Scott Aaronson wrote an insightful paper about this.

All formal sciences

Formalism has expanded far beyond metamathematics and theoretical computer science.

Many fields now have their formal variants, like economics and game theory, or biology and computational biology.

Most fields use formal techniques, whether it is model-based computer simulations or just copious amounts of data science.

New fields have also been created that are exclusively formal, like programming, automated theorem proving or cryptography.

We now are fish in the water. We live in the formal era. Formalism is everywhere, and it feels like normal.

It is hard to imagine a pre-formal world, where people did not immediately understand why it is possible to write a program interpreter, when there are so many online tutorials anyone can use to create their very own programming language, let alone merely interpreting an existing one like LISP.

It is hard to imagine a pre-formal world, when people were confused about what made a proof rigorous, and when geniuses made mistakes that a smart high-schooler would immediately notice.

The Spirit of the Formalist Arc (into the future!)

Doing Things Well

For me, this arc of human history is about something deep.

It is the result of times when geniuses tried to do things thoroughly.

When these maths geniuses noticed that notations and maths worked really well, they didn’t go “I guess it’s just that maths is good lol”.

They took what felt like a non-scientific question, the nature of maths, and turned it into a scientific question.

They tried to understand what really made maths work, reverse engineer it, and completely solve it.

And they did! In a few decades, they unlocked an entire new section of the technological tree. They created a large part of the modern world! Automation and information technologies are directly downstream of their effort.

—

Nowadays, our geniuses seem very meek, content to just explore their very specific niches and to profit from systems which they do not understand. I believe they should instead try to unravel the crucial mysteries of our times.

In the past, many Greeks thought that the stars were divine beings or heroes immortalised in the heavens, that natural disasters were caused by the gods, or that Chaos birthed Gaia. And we transcended these beliefs!

There are crucial topics where we are as lost as mathematicians were in the past, and where we should try to pull what we pulled with formalism: a definitive answer to these deep questions.

Right now, these crucial topics might seem non-scientific, but the history of science (including formal sciences!) has been a long conquest of what was thought to be the realm of metaphysics and supernatural entities.

Governance

The Enlightenment was great. Many great thinkers sought to come up with a alternative to kingdom that was better for the people, and they largely did.

This led to a governance explosion in the 18th and 19th centuries.

Sadly, since then, we have mostly stalled. Our governments are still using the Separation of Powers in three branches (legislative, executive and judiciary) that Montesquieu spontaneously conceived in 1748.

On the theoretical side, we have done a lot! We have developed game theory, public choice theory, voting theory and even distributed consensus algorithms.

And yet. Very little improvement in governance.

—

The biggest example of this stagnancy is voting.

Voting every couple of years was the best that our ancestors could do with the technologies of their time.

A time when the fastest ways to transmit information were homing pigeons and horse carriages! When countries had much less transportation infrastructure like railways or airports. When people were much less educated and knowledgeable than they are now.

We now have the Internet, which lets us transmit information as fast as physics will let us and most people graduate from high school.

But we still use anachronistic voting systems. This is ridiculous!

—

There are so many problems in governance.

How can we integrate experts into policy making, with proper checks and balances to avoid academic capture and the politicisation of science?

How can we hold politicians accountable for their lies, and ensure they take action when it matters?

How can we reliably deal with negative-sum games?

How can we deal with entrenched interests, special interests and concentrated interests?

It seems quite sad to me that we are not any closer to a solution on any of them than we were 20 years ago.

But let’s move to another topic close to my heart, where we also need a reckoning, similar to maths’ foundational crisis.

Morals

Our morals have changed a lot in the recent decades, and a fair share of it can be qualified as moral progress.

Sadly though, the vocabulary of morals has not changed much. We can find most of it in the Universal Declaration of Human Rights from 1949.

To be clear, there have been a couple of moral innovations, like Data Rights, Privacy Rights, Indigenous Rights as well as Sustainability for the Future.

But at large, it doesn’t feel like we made that much progress on identifying Human Values and good ways to trade them off. We are still thinking in the same terms, through the same types of principles, many that can be traced to 18th century and have been reused as is.

—

I don’t expect that debating morals with my contemporaries would make me so much smarter than debating with people from the 1950s.

It looks like we are still as confused as we were before.

This is in sharp contrast with technical topics.

I expect that discussions on any technical topic will be much richer in 2025 than in the 1950s, and that any scientist from back then would love the opportunity to have access to our current body of knowledge.

It is a shame that this is not the case for morals.

More!

There are so many more important topics where we are thoroughly confused.

Psychology and Sociology. How do people work? How should we reason about people? How should we predict our own behaviour?

Meditation and Introspection. It looks like we can get many benefits from thinking about the correct stuff. Like making better decisions, having more control over our attention and curiosity, being calmer and less stressed, learning faster, and feeling better. Yet there is no scientific body of meditation/introspection techniques that I can use. It’s usually a bunch of spiritual new-age mumbo-jumbo. Aaaah!

Management. What are the best ways to work together? In non-profits, for-profits, in families and in friend groups, I have always seen people use a combination of natural talent, experience and heuristics. We ought to build a science of management that actually lets us build better organisations!

What can we learn?

I believe there are a few lessons we can derive from the Formalist Arc.

1. Lots of work

The first one is that solving formalism required a lot of concentrated work from a lot of geniuses.

Instead of working in finance, SaaS, GenAI or ad-tech, they were working on this.

Instead of maximising their impact factor or publishing one more paper they did not believe in, they were working on this.

This took sustained work over decades from the brightest of their time.

Where are today’s brightest spending their time?

—

Personally, I find it quite sad that the brightest people I know are not trying to thoroughly and scientifically solve the problems I mentioned.

I find it sad that many play too much video games or waste their talent on questions that are too narrow and irrelevant.

It takes a lot of work from very driven geniuses to solve such problems, and we are not harnessing the ones we have.

We should be planning for many such projects. We are not lacking in important problems that we must solve.

I think there is just a big mismatch, and that the mismatch is mostly explained in this article. (Tl;dr: what people spend their time on is rarely the result of a deliberate process, and this is especially true for geniuses.)

2. Lots of ambition

The second one is that ambition is crucial.

Many of them sought to fully solve maths. The most emblematic of which was Hilbert’s Programme.

This was not vain ambition. Hilbert’s Programme was specifically about designing methods that could in principle solve maths.

This was not intellectual masturbation either. Hilbert’s Programme was so meaningful that rejecting it was enough to usher in the formalist era.

It was a serious ambition, with a solid plan to solve The Big Problem. The plan was good, and it needed to be good such that failure would also be informative.

—

In contrast, I don’t know of an equivalent of Hilbert’s Programme in any of the fields I have outlined earlier.

I simply believe that despite all of the hype around getting rich, startups and VCs, we are just much less ambitious.

“I will make a gazillion money by building the next Netflix or Uber” seems so much less ambitious than “I will meaningfully contribute to an understanding maths so thorough that it will affect Humanity’s entire tech tree”.

“I will cut a lot of government spending” seems so much less ambitious than “I will solve governance and the solutions will be so good that both politicians and governments will be forced to pick them to stay competitive”.

It might be that we can’t replicate the methods that let us solve maths in governance or morals. But I don’t care much for this thought. Until we try the Obvious Thing and fail, I don’t think we ought to give much attention to pessimists and those who say we can’t do things.

—

To be fair, I believe many people are simply afraid of being too ambitious. And the fear makes sense to some extent.

Some people are so obsessed with power that they are willing to hurt others over it. They will trample over everyone else, and do things that they would consider immoral if anyone else was to do the same.

Even worse, most people who are too ambitious are just untethered to reality. They become megalomaniac, narcissistic or even the “I will be the Second Coming of Christ” type of people.

However, I think that it is possible to cultivate a good type of ambition.

The type of ambition where we contribute toward a critical intellectual project in a grounded manner.

The type of ambition where we want to help and strengthen as many people as possible.

This story is a nice way to internalise what this good type of ambition looks like.

3. Long Lists

The third lesson is that lists are good.

Too many people who want to work on a crucial problem try to assume its complexity away.

—

I had a recent discussion about governance. My interlocutor then told me about their economist friend who had an ideal model of [something] and I sadly sighed.

I do not care much for ideal models. Not because they can’t be useful (I’m a formalist, of course I think ideal models can be useful!).

Just because people who work on ideal models tend to be the type of people who will do anything but engage with the real world. They will never be people who tried to implement better governance principles in an organisation, failed, and are now tackling the bottlenecks to doing it.

They will be people who find talking about these specific ideal models interesting.

What they will never do is write a long list of desiderata and constraints that they got from the real world, which their models should satisfy.

—

Hilbert’s Problems were a great example of this.

Even maths, the purest field ever, was big and messy. And Hilbert embraced the mess. 23 problems, all from different subfields and of different natures, meant to surround maths from every angle.

With his understanding of the time, he put down in writing what he thought were the most relevant problems of maths, with various precision levels.

(To be fair, I’d say he was still too shy! His 24th problem, which he did not publish, was the best one.)

—

I want Hilbert’s Problems of Morals, of Governance, and of everything!

How can a field be considered a serious field if I cannot even see its top 20 open problems? If I cannot see them, that’s because scientists are not constantly tracking them.

This in turn reveals that they are not collectively looking at a clear North Star. In other words, there is no big picture. They are much too lost in short-term fads and turbulences.

If I am to consider a field seriously, at the very least, it should have a list.

Conclusion

I love the story of formalism for many reasons.

Some are objective: I truly believe it is one of the core insights behind modern science.

Some are more personal: I investigated formalism by myself quite a lot, before learning about it from others.

But the biggest reason is that it is inspiring.

Through sheer collective intellectual genius, we solved one of the most metaphysical questions (“Why does maths work?”).

We worked at operationalising it for decades until we could solve it.

And then we did solve it. The response we found was unexpected, and so important that much (most?) of our modern world relies on it.

I think we should do more of that.

—

I did not mean for this article to be that long. Sorry!

But this is a story that I love telling orally, and I must get better at writing down my stories.

So here it is!

On this, cheers, and have a nice day!