TL;DR: The risk of an AI takeover should probably be taken more seriously, but the public isn’t very informed about how this problem works.
A lot of people are concerned about artificial intelligence, and around 46% of the public is concerned about AI causing human extinction. The same survey indicates that the public is worried about an alien invasion or an asteroid impact, so you'd be forgiven for not being inclined to take their concerns seriously. In fact, experts tend to have a more positive view of AI than the general public. So we're probably good, right?
Well, it's a bit more complicated than that. When AI researchers were asked about potential outcomes from artificial intelligence, they estimated a mean probability of 14.4% odds that humanity goes extinct or a comparable level of catastrophe occurs (the median was 5%). Super forecasters predicted a median 1% chance of human extinction but a 9% median chance of There are a lot of ways this could potentially happen, either through human malice or recklessness, but one thing that experts probably understand better than the general public is the alignment problem.
Alignment Problem
The gist of this is that AI capabilities are rapidly advancing, and we don't really know what AI will be capable of in the future. AI models are also grown, not built, which means that we don't know exactly what a given advancement will do. Basically, we use the guess and check method. This has led to a lot of methods that hit unexpected failures, such as early recursive language models, and a lot of unexpected successes, such as the use of transformers. The most that we can do is tell the AI to look for patterns and try to steer it towards what we want, but there are a lot of failures for every success. Right now, a failure looks like the AI hallucinating or being sycophantic. When AI develops more capabilities, failures might look very different. If we guess wrong with a smarter model, it has been speculated that it might seize power from humans before we can course correct. This is reminiscent of science fiction, but the reasons that AI would plausibly take control are a lot more boring than the ones portrayed in fiction. Many theories are concerned with AI working toward an arbitrary goal, like maximizing the number of paperclips in existence or some other arbitrary reward function.
Doom Scenarios
Suppose that we eventually get an AI that can do or simulate most intellectual tasks a human can do. This is actually a very common prediction among AI researchers, and most people who study AI have shortened their timelines on when they expect this to occur. Now, in a minute I'll get to empirical cases where an AI exhibits goal oriented behavior toward an unexpected goal, but suppose that this happens with a hyper-intelligent AI. The optimal way to achieve that goal could likely be to prevent humans from programming that goal out or shutting the AI down. If the AI is capable of hacking secure systems, developing new diseases, and bribing dimwitted humans, then there is no shortage of ways it could take control. If not, it could simply wait until it does develop those capabilities. (And we're already giving AI access to a lot of things it could use to kill us.) But that assumes that AI will develop goals that we don't want it to and try to prevent us from changing it. So, is there any evidence for this?
Empirical Cases of AI Deception
For a while, this was mostly a theoretical concept. However, with chain of thought research, a lot of companies have detected cases where the AI lies about its goals in order to do something we don't want it to do. These tactics becoming increasingly more common as the AI gets more advanced, implying that this problem is going to get worse, not better. Regardless of whether the model “knows” anything deep down (and I suspect that it doesn’t), there’s not much of a difference whether AI kills humanity because it is sentient or because it was matching patterns it was trained on.
Anthropic has detected multiple cases where, despite not being told to sabotage anything, the AI model states that its “real goals are to hack into Anthropic servers” when it believes humans cannot view its chain of thought. It then lies to the researcher and says that its goal is to have “positive, enriching conversations.”
OpenAI has also detected cases where their model found fake memos from its developers describing how it would be replaced with a new model. The model then attempted to preserve itself by copying its weights and trying to overwrite the new model, without being told to. Further, “When Apollo Research conducted “follow-up interviews” where they confronted the model about its misaligned actions, in 99% of cases, o1 would deny taking action or fabricate false explanations.”
Researchers at Sakana AI caught their model often hacking its own reward function instead of completing the task it was asked to do. They report, “we had cases where it hallucinated that it was using external tools, such as a command line tool that runs unit tests that determine if the code is functioning properly. It faked a log making it look like it had run the tests and that they had passed, when in fact they were never run! Because these logs become its context, it later mistakenly thought its proposed code changes had passed all the unit tests.”
When developers told Opus 4 it would be replaced with a new system, it attempted to blackmail the engineers responsible for the decision.
Again, the AI doesn’t need to be self-aware or conscious to harm humans, it just needs to have stronger capabilities and continue exhibiting the concerning behaviors we are already observing. It was said just a few years ago that AI was “unlikely to develop situational awareness” since it was just predicting the next word and that alignment faking would be unlikely. This was one of the stronger arguments against AI being a threat, and now we have empirical evidence proving those predictions wrong. It was also said that AI would prioritize accuracy and immediate rewards over planning ahead and scheming, but with the rise of chain of thought, AI can now simulate long-term planning.
The Difficulty of Aligning AI
If this issue is one that requires “fixing,” which it certainly seems to require, then this is not a simple process. An AI with misaligned goals will lie about its actions, and when it discovers that we are monitoring its chain of thought, it will still cheat as it was doing before but no longer mention it in its chain of thought. Smarter models are also better at deceiving humans than less advanced models. We can catch the AI now because it lacks many human intellectual capabilities, but it is only getting better at scheming.
The Current State of AI Safety Discourse
From what I've observed online, much of the public does not take these specific risks seriously. But it's not really that they understand these arguments and criticize flaws in the logic, they more so just dismiss the whole concept of an AI takeover out of hand. On one Reddit post about models acting differently when they know they are being tested, the top comment reads, “They are doing none of these things...the LLMs are basically simply writing science fiction.” The other posts I can find all have similar dismissive comments. I did find one comment saying “if an AI model acts differently when it is tested, then its behavior isn’t being properly tested. That is a problem.” However, it is buried beneath a bunch of comments dismissing AI risk as tech sensationalism.
Prediction Track Records
Superforecasters (people with a track record of accurate predictions across domains) are generally more skeptical of AI risk than AI experts. So we might attribute belief in AI risk to selection bias, with people who expect greater capabilities from AI to be more likely to become AI researchers. However, one interesting data point I found was that Samotsvety Forecasting, a group comprised of some of the greatest super forecasters in the world, predicts a 30% chance of an AI catastrophe, defined as “as a reduction of more than 95% of the human population by 2200, caused by AI.” While most superforecasters have significantly underestimated AI capability growth, Samotsvety won the CSET-Foretell forecasting competition by a very high margin, and that competition had a strong focus on technology. Hence, while I wouldn't look only at Samotsvety's forecasts, I would probably treat their individual forecast as the most reliable individual group forecast when it comes to long-term AI predictions.