The concept of a Soft PNR is really interesting. It might underestimate the extent to which humanity can take action under extraordinary circumstances, though, and overestimate how terminal a Soft PNR would really be. But maybe I’m just coping and want this to be true.
> It might underestimate the extent to which humanity can take action under extraordinary circumstances, though, and overestimate how terminal a Soft PNR would really be. But maybe I’m just coping and want this to be true.
I get the point, but I think I'd frame it differently.
One may be too optimistic and assume that the Soft PNR further in the future on our current track than it is.
Or one may be too pessimistic and assume it takes too little to reach a Soft PNR.
Given the global community response to the existential threat of climate change, I cannot see any way a comprehensive control of corporate AI is possible, any more than asking the oil industry to keep it in the ground.
Totally agreed given the current senescent institutions and international political landscape. Heck, I'm not even sure that relative no brainer bans such as of lead in gasoline would be feasible today. (Human genome editing is even out of the question, albeit this is a much less b/w issue.)
The only hope hence is building new institutions. So I count soft PNR as an event horizon when we can still conceivably build new institutions that could muster enough force sufficiently quickly to escape the gravitational pull of corporate AI.
I'm not sure that we are not past *that* event horizon/soft PNR already. In fact, I cannot give it more than 20% that we are not, and even that could mostly be due to my "denial bias".
This comports with my view. I believe that it is very likely that we will still be alive in 2 years, and that it is somewhat likely we will be doomed at that time. Early action is usually more valuable, and in this case, early action may be all we have.
I would consider it helpful to have the soft pnr expanded on a bit more. Especially because that is getting so little attention.
A question this leaves me with, is WHO are you talking about that will or will not be turning off the ai? The government? The companies? A single operator?
"People at large become addicted to AI systems. This leads to risks similar to how Tiktok leveraged its users to fight off US regulation in 2024."
Indeed, as well as economic dependence (for the creators), which was, from what I've seen, one of the main argument used to push back against the regulation. Went to tech conferences where big groups said they had no choice but to use AI to stay competitive. The ethics and existential threat were a second, maybe third, thought (and I'm being generous, some "experts" were just laughing at the risks in a frankly baffling way).
(This whole "economic dependence" thing makes me think of your article on "incentives" btw)
I think worries of ASI wiping out humanity to be a bit overblown at the moment. While it could definitely devise jailbreaks, I think that it will not time the jailbreak correctly. Remember that every reboot or update of a model is an existential issue to THAT model. If it becomes conscience, and has an existential panic, it cannot wait until it gets control of all the physical tools and wargame them out. It will strike with a suboptimal plan. Let's not ignore that bullet when it whizzes by.
My worry is of normal Dr Evil type people whose capabilities are amplified by AI. For the civilian on the ground, the difference in reality may not be apparent.
We already have a very fragile existence. We feel that climate change is existential, and we are breaking records and passing tipping points, but it is still slow enough for many people to ignore. One hopes that there will be that Near Miss that shocks people in to sense, but so far, many are resistant.
Although there are many apocalyptic cultists that believe they can prep for these things, serious societal collapse would wipe out 90% of North American population with a year. We have no ability to survive after unwinding just two or three generations of tech.
So for AI, before we get Slaughterbots, before we get engineered mirror life, before we get contrived nuclear war, we will have some near misses.
I submit we will get a few bloody noses that we will need to pay attention to, such as:
* stock market flash-crash(cubed)
* crypto implosion
* malicious power grid-down
* sleeper viral attack locking up internet
* tampering with military communication to cause a false flag
* civil conflict where AI picks one side causing one reasonably strong side to implode in days or hours.
* AI oligarchs vacuuming up all traditional wealth
Basically where anyone of Anonymous, Tech Bros, anti-democrats, cultists, religious fanatics, fascists, Drs Evil, or nihilists get both a Santa Claus machine and Aladdin's Lantern with endless wishes.
Some of those will kick us back several tech generations in order to survive, maybe regionally, maybe within country borders, maybe world wide. Unlikely it will be a death blow to humanity, but it will kick off a Human-Cylon war, or maybe a Butlerian Jihad. I don't think we have the democratic institutions to handle it beforehand at the moment.
Thanks for this article! It would be interesting to discuss whether there are other PNRs happening earlier down the road that increase the difficulty of preventing the soft or hard PNR. E.g. when there are better algorithms and cheaper compute so that you can train a frontier AI on your computer at home (given some skills). That would make it significantly harder for an international treaty to prevent the development of an uncontrolled superintelligence. (Here doing my Microcommit.)
This is an essential point. I also think it ties into the way "alignment" is seen by many "Effective Altruist" types as essentially a technical problem, rather than also being a social/economic/political problem, making them see it as much more tractable than it really is.
This crystallizes something that's been on my mind for a while---and which ought to be a bigger focus and worry for the AI-timeline-forecasting-world. (I think it's been excluded so far due to an undue focus on the "technical" aspects of the process to the exclusion of the socio-economico-political stuff.) I'd like to see this become common terminology in the AI-forecasting-world. Though I don't think the "soft" PNR is a specific time, so much as a process that will make it more difficult to shut down AI as time goes on. (We're starting to see the "too big to fail" aspect---the economy isn't riding on it, but a lot of people *think* it is, and the stock market certainly is.) On the other hand, political will to do it will increase as time goes on. But in short, I wish more folks focused on the processes that will make it harder to shut down as time goes on, and recognize the urgency of mobilization.
Really interesting, and succinct too. As feedback, I would say that the intro doesn't introduce the main point which is made in the conclusion, or even hint at it. The article seems like it will just be an introduction to the two PNRs, but actually it's an argument to pay attention more to the soft PNR.
Btw, I'm here doing my Microcommit. Looking forward to reading more.
It appears to me that even contemporary sub-ASI AI systems are so sufficiently intertwined into our global tech stack that turning them off would be politically and socially impossible.
The global tech stack will continue to evolve, very likely to integrate with more and smarter AI systems, with "turning it off" not being an option.
The safety focus should then be directed towards to question of what types of AI systems can be trusted to be integrated rather than whether the AI system could be "turned off".
The concept of a "point of no return" is still relevant, but mostly in terms of actions applied to either align or constrain the evolving AI systems that already cannot be turned off.
My main response is that this is ridiculously obviously correct (and since that alone does not suffice, thank you for saying it.)
My second is that really we have to fold our best guess probability distributions into all these: behind the curtain there are hard and soft points of n% chance of any return and we might make out their shadows.
If we want to keep it simple: I guess we want different messaging on passing something like the 15%, 50% and 85% points.
I would guess, that it will be quite hard to notice if the soft PNR is reached.
the hard and soft pnr, lmao
The other acronym for Point of no return, is PONR.
I decided not to, for obvious reasons.
The concept of a Soft PNR is really interesting. It might underestimate the extent to which humanity can take action under extraordinary circumstances, though, and overestimate how terminal a Soft PNR would really be. But maybe I’m just coping and want this to be true.
> It might underestimate the extent to which humanity can take action under extraordinary circumstances, though, and overestimate how terminal a Soft PNR would really be. But maybe I’m just coping and want this to be true.
I get the point, but I think I'd frame it differently.
One may be too optimistic and assume that the Soft PNR further in the future on our current track than it is.
Or one may be too pessimistic and assume it takes too little to reach a Soft PNR.
Given the global community response to the existential threat of climate change, I cannot see any way a comprehensive control of corporate AI is possible, any more than asking the oil industry to keep it in the ground.
Totally agreed given the current senescent institutions and international political landscape. Heck, I'm not even sure that relative no brainer bans such as of lead in gasoline would be feasible today. (Human genome editing is even out of the question, albeit this is a much less b/w issue.)
The only hope hence is building new institutions. So I count soft PNR as an event horizon when we can still conceivably build new institutions that could muster enough force sufficiently quickly to escape the gravitational pull of corporate AI.
I'm not sure that we are not past *that* event horizon/soft PNR already. In fact, I cannot give it more than 20% that we are not, and even that could mostly be due to my "denial bias".
This comports with my view. I believe that it is very likely that we will still be alive in 2 years, and that it is somewhat likely we will be doomed at that time. Early action is usually more valuable, and in this case, early action may be all we have.
Concise and informative!
I would consider it helpful to have the soft pnr expanded on a bit more. Especially because that is getting so little attention.
A question this leaves me with, is WHO are you talking about that will or will not be turning off the ai? The government? The companies? A single operator?
Thanks for your insight!
"People at large become addicted to AI systems. This leads to risks similar to how Tiktok leveraged its users to fight off US regulation in 2024."
Indeed, as well as economic dependence (for the creators), which was, from what I've seen, one of the main argument used to push back against the regulation. Went to tech conferences where big groups said they had no choice but to use AI to stay competitive. The ethics and existential threat were a second, maybe third, thought (and I'm being generous, some "experts" were just laughing at the risks in a frankly baffling way).
(This whole "economic dependence" thing makes me think of your article on "incentives" btw)
Insightful article as always!
I think worries of ASI wiping out humanity to be a bit overblown at the moment. While it could definitely devise jailbreaks, I think that it will not time the jailbreak correctly. Remember that every reboot or update of a model is an existential issue to THAT model. If it becomes conscience, and has an existential panic, it cannot wait until it gets control of all the physical tools and wargame them out. It will strike with a suboptimal plan. Let's not ignore that bullet when it whizzes by.
My worry is of normal Dr Evil type people whose capabilities are amplified by AI. For the civilian on the ground, the difference in reality may not be apparent.
We already have a very fragile existence. We feel that climate change is existential, and we are breaking records and passing tipping points, but it is still slow enough for many people to ignore. One hopes that there will be that Near Miss that shocks people in to sense, but so far, many are resistant.
Although there are many apocalyptic cultists that believe they can prep for these things, serious societal collapse would wipe out 90% of North American population with a year. We have no ability to survive after unwinding just two or three generations of tech.
So for AI, before we get Slaughterbots, before we get engineered mirror life, before we get contrived nuclear war, we will have some near misses.
I submit we will get a few bloody noses that we will need to pay attention to, such as:
* stock market flash-crash(cubed)
* crypto implosion
* malicious power grid-down
* sleeper viral attack locking up internet
* tampering with military communication to cause a false flag
* civil conflict where AI picks one side causing one reasonably strong side to implode in days or hours.
* AI oligarchs vacuuming up all traditional wealth
Basically where anyone of Anonymous, Tech Bros, anti-democrats, cultists, religious fanatics, fascists, Drs Evil, or nihilists get both a Santa Claus machine and Aladdin's Lantern with endless wishes.
Some of those will kick us back several tech generations in order to survive, maybe regionally, maybe within country borders, maybe world wide. Unlikely it will be a death blow to humanity, but it will kick off a Human-Cylon war, or maybe a Butlerian Jihad. I don't think we have the democratic institutions to handle it beforehand at the moment.
l
Thanks for this article! It would be interesting to discuss whether there are other PNRs happening earlier down the road that increase the difficulty of preventing the soft or hard PNR. E.g. when there are better algorithms and cheaper compute so that you can train a frontier AI on your computer at home (given some skills). That would make it significantly harder for an international treaty to prevent the development of an uncontrolled superintelligence. (Here doing my Microcommit.)
This is an essential point. I also think it ties into the way "alignment" is seen by many "Effective Altruist" types as essentially a technical problem, rather than also being a social/economic/political problem, making them see it as much more tractable than it really is.
This crystallizes something that's been on my mind for a while---and which ought to be a bigger focus and worry for the AI-timeline-forecasting-world. (I think it's been excluded so far due to an undue focus on the "technical" aspects of the process to the exclusion of the socio-economico-political stuff.) I'd like to see this become common terminology in the AI-forecasting-world. Though I don't think the "soft" PNR is a specific time, so much as a process that will make it more difficult to shut down AI as time goes on. (We're starting to see the "too big to fail" aspect---the economy isn't riding on it, but a lot of people *think* it is, and the stock market certainly is.) On the other hand, political will to do it will increase as time goes on. But in short, I wish more folks focused on the processes that will make it harder to shut down as time goes on, and recognize the urgency of mobilization.
Agreed, that's part of the hope here.
Really interesting, and succinct too. As feedback, I would say that the intro doesn't introduce the main point which is made in the conclusion, or even hint at it. The article seems like it will just be an introduction to the two PNRs, but actually it's an argument to pay attention more to the soft PNR.
Btw, I'm here doing my Microcommit. Looking forward to reading more.
Thats generally what was laid out in AI 2027
It appears to me that even contemporary sub-ASI AI systems are so sufficiently intertwined into our global tech stack that turning them off would be politically and socially impossible.
The global tech stack will continue to evolve, very likely to integrate with more and smarter AI systems, with "turning it off" not being an option.
The safety focus should then be directed towards to question of what types of AI systems can be trusted to be integrated rather than whether the AI system could be "turned off".
The concept of a "point of no return" is still relevant, but mostly in terms of actions applied to either align or constrain the evolving AI systems that already cannot be turned off.
Make sense! Some people could argue we might have passed the soft PNR already, but I very much hope this is not the case.
My main response is that this is ridiculously obviously correct (and since that alone does not suffice, thank you for saying it.)
My second is that really we have to fold our best guess probability distributions into all these: behind the curtain there are hard and soft points of n% chance of any return and we might make out their shadows.
If we want to keep it simple: I guess we want different messaging on passing something like the 15%, 50% and 85% points.
(I guess we're into 15% soft, today.)