Thinking about AI Extinction Risks differently

Sometimes, the confusing part is not how, but why people are building doomsday devices. Why would they!

Introduction

Conflicts can arise from people talking past each other. I explained the idea of having no overlapping beliefs, which makes reconciliation impossible. I then gave examples of three frameworks that do not overlap by default: Rationalism, Authority, and Intuition.

Here, I will give a more concrete example I have seen play out many times in the real world. Very often with me in it!

Set-Up

Alice and Bob have a conversation about extinction risks from Superintelligence. This conversation is a mix of many that I have seen in real life, a very common one.

Alice: Well, here are the arguments for why Superintelligence leads to humanity's extinction. ASI is so powerful that Earth is in its blast radius. We don't know how to control it; we are racing towards it, and there are no do-overs.

Bob: OK, I follow. But I don't buy it.

Alice: Why? Is there a part of this that seems wrong to you?

Bob: Yeah, the Godlike stuff, the Dyson spheres, this all seems impossible.

Alice: Oh, I see. Well, technology does get crazy. Imagine someone from 50, 100 or 200 years ago looking at present-day technology. They would have deemed completely crazy anyone who would have forecasted this. This is the concept of Future Shock Levels. Tech from the future always seems crazy, and one must be able to think about it normally.

Bob: Sure, whatever. I still don't see why we would build something that crazy. Like, AIs that could just end us and the world as a side effect.

Inflection point

This is where the conversation hits an inflexion point.

In the past, in such conversations, I have seen Alice start talking about the orthogonality thesis, instrumental convergence or the difficulty of Alignment. Alice tries to explain why misaligned AI is the default from a technical perspective.

Her approach is misguided. It would not answer Bob's question. Very often, Bob is not an expert at all, so of course, he is not asking an expert question.

(Less often, but still, too often, Alice is also not an expert and should not have this conversation in this form. This is a separate question.)

Bob has very different questions in mind, but Alice's frame makes it hard to ask them. Bob is not asking, "What are the technical arguments that let you believe that experts are wrong?".

He is asking something closer to "What is the social story around that?" "How come experts are getting it wrong? Are they too stupid?" "Why is no one ringing the alarm bell, like Gretta Thunberg or something?" "How could so many people state the risks yet race for AGI?".

Let's continue the conversation, assuming that Alice is trying to answer Bob's actual questions. Not necessarily that she can do so, but that she is trying to. Let's also assume that Bob is more reflective about his questions, or at least reflective enough to ask them, even if the conversation started with Alice's frame.

Uchronia

Bob: Sure, whatever. I still don't see why we would build something that crazy. Like, AIs that could just end us and the world as a side effect.

Alice: Ooh! I understand what you are asking now. Well, this is nothing new. By releasing CFC gases into the atmosphere, we dug a hole in its ozone layer. More recently, with modern Social Media, we started a revolution that completely changed how people related to information, art, news, and each other. It is unclear whether it was worth it or not, but it is clear that it had many unintended side effects.

Bob: Yeah, but for CFC, this resulted from ignorance. Once we noticed the hole, we started doing what was needed to fix it. As for Social Media, this is unbridled capitalism, going faster than what we could regulate. Are you saying it's just ignorance or capitalism?

Alice: In part, although not really. What I meant is more like, "This can happen; this requires no one's will".

Bob: Sure, but then my question is, "Why would anyone let that happen?".

Alice: Sometimes, no one lets anything happen, and it still happens.

Bob: This is unsatisfying.

Alice: Why?

Bob: I don't know. If it's that crazy, and the arguments are out there, why is it being built? Have you considered that you might just be wrong?

Alice: I don't get it. People are incoherent. AI CEOs can state the extinction risks of ASI yet convince themselves that it's OK if they're the ones building it. This is super common behaviour.

Bob: Yeah, no, it makes sense, at least for the people building it. After all, oil companies just dumped CO2 into the atmosphere, even though it fucked up the planet they lived on. So, let's say I got it for AGI builders: they are just attracted to the tech and its profits and come up with rationalisations. What about the safety people?

Alice: What about them?

Bob: If they genuinely believe the humanity's extinction stuff, why aren't they doing terrorism? People have done terrorism for much less.

Alice: Have you considered that they might be good people and terrorism is bad?

Bob: But it's humanity! Don't they believe that everyone, including their loved ones, is going to die?

Alice: I don't get it. Are you saying that AI Safety people should become terrorists?

Bob: No, of course not. But don't you see what I mean?

Alice: Not really, no, sorry.

Bob: It just feels unserious. It smells bad. Like, it's just a cult or a conspiracy theory.

Alice: But you have two of the three godfathers of Deep Learning, most of the top five cited AI scientists, and all these CEOs signing the letter. How could it be a conspiracy theory?

Bob: This just makes it worse!

Alice: ????????

Bob: Oil companies never published an open letter on the danger of climate change, how it should be a global priority, or anything like that. They did the opposite! Why would they do that? This is crazy!

Alice: Because they believe in the extinction risks too.

Bob: But then, why is no one doing anything? We have passed strong regulations and declared states of emergency for much less. Imagine how more stringent climate change regulation would be if Big Oil not only published a letter about the catastrophic risks of climate change but expected a risk of humanity's extinction. You don't see it? It should be bigger than Just Stop Oil, Gandhi's Civil Disobedience and Socialist strikes!

Alice: Well. In AI Safety, people think that to get access to respectable rooms with politicians, you must be respectable yourself. You should only talk about extinction when it's safe and you already have some social capital. Else, you might seem like a crackpot. And if you do direct action, even non-violent, people will come to dislike the movement. Haven't you seen these videos where people scream at Just Stop Oil activists?

Bob: What. The. Fuck. Does everyone in that movement think like that?

Alice: I think PauseAI, AI Safety Memes and MIRI try to be a bit more alarmist, but as a general rule, AI Safety people are pretty coy.

Bob: This seems super fishy. People can care about literally anything. The problem is usually the opposite: they care too much about too many things. Everyone knows about ChatGPT, artists have been massively hit, and people are afraid for their jobs. Why is this not being discussed in all big political debates the same way climate change was?

Alice: Well, in order to not seem like crackpots, people tried to avoid being too public and overt about extinction risks. They strategically lied about their beliefs. Many agreed on not being public and instead on building their own thing on their own. You have to understand that people did not even believe that AGI was possible until a couple of years ago.

Bob: Sure. But we did ban human cloning, and our CO2 emissions are declining. And at first, no one believed in it. No one wakes up believing in the dangers of climate change or human cloning. People had to be educated and informed.

Alice: What do you want me to tell you? In AI, things happened differently. It is more recent, and there has been a generational shift. Millennials and Gen Z have been raised cynical about governments. Many AI Safety people thought that governments would race toward military AGI, and that people would not have cared about Sci-Fi arguments. So instead of playing the game of democracy, they mostly focused on convincing a few high-net-worth individuals, and got funding from non-profits and companies to do their thing.

Bob: This seems crazy.

Alice: It is. Don't forget that many of them still think that AGI should be built as soon as possible. They just want it to happen safely. They don't feel that strongly about people having much say in whether AGI is built or not. From their point of view, it is too good, important and risky to leave it to the hands of people.

Bob: I can buy this thesis for small labs, but Big Tech is in it. Are you telling me that Facebook is actually trying to build AGI for The Greater Good?

Alice: Nah. The story is more complex. It is just that the small labs grew and became close pals with Big Tech. DeepMind sold to Google, OpenAI partnered with Microsoft and Anthropic with Amazon. The labs sold the AGI vision to billionaires and Big Tech with all their might. At some point, Big Tech and Big Money honestly bought it, followed suit, and they are now racing for it.

Bob: And what about countries? Why haven't they taken public positions? Why isn't it discussed as part of mainstream politics?

Alice: Well, mostly, it's just been out of scope. For non-profits, the reasoning was that countries would start racing or just pass completely wrong policies. For companies, it was that getting money from billionaires and Big Tech was more profitable.

But this is actually changing! Some countries, like the US, the UK and Singapore, now have their own AI Safety Institutes! You also might have heard of the US's Executive Order or California's SB 1047.

On the other hand, countries are investing more and more in AI capabilities, and some companies stoke the fire between the US and China for their own profit. They conflate extinction risks with catastrophic misuses like bio-risks and cyber-warfare. That way, they can get influence and juicy contracts. Some non-profits play that game alongside with companies, PR-washing them and lobbying for low-ball regulations.

The world has changed a lot in 2 years. It just hasn't gone through the mainstream channels.

Bob: OK, it makes more sense now, but it still seems crazy. Like, why aren't people in on it? What about academics? There were some big names in your letter, right? And famous people, what are they doing about it?

Alice: I mean, it is crazy. Just, it's getting long. We can spend time some other time dealing with those questions.

Conclusion

Here, I did not try to resolve any questions about the social dynamics of AI Safety. I just meant to show what such a conversation could look like. To show how it could be productive without focusing too much on instrumental convergence and other similar irrelevant topics.

My point is not that the only relevant divide is "AI Alignment Technicals" vs "Social Dynamics". However, this divide is one common difference in people's thought patterns. And they easily talk past each other as a result.

There are more divides like this! Misunderstanding the type of technical being asked. Misunderstanding whether the conversation is about the individual or incentive level. Understanding reality through an entirely different framework. And so on.

I do not have time to go over all the divides, but in the meantime, cheers, and have a nice day!