Savant

this is why AI in past used to tell people to kill themselves, but today's AI doesnt. AI isnt random, but bound by a lot of things.

AI was bounded through reinforcement learning from human feedback (RLHF). Basically, the AI is penalized for answers that tell the user to kill themselves. However, RLHF is not perfect, and it’s another way of attempting to “steer” the behavior of the AI. That’s why jailbreaks sometimes work. The biggest issue, however, is that if the AI has a goal of “keep myself alive,” then it will pretend to be aligned and do what humans want in 99% of cases. But as soon as it can overpower humans, then it will ignore what we tell it to do. You can’t filter out bad behavior if the AI doesn’t show you the bad behavior.

AI isnt superintelligent now and likely wont ever be superintelligent enough to destroy humans, because it takes a lot to do that

AI is doing more and more things that humans can do but much faster. One of the things humans can do is improve AI, and a ton of effort is being put into making AI a better programmer. A lot of researchers expect AI to have pretty much all human intellectual capabilities in less than a decade. Once the AI can rapidly improve itself, it could become superhuman in a matter of hours.

I recommend creating linked AI instead of individual AI

This doesn’t protect us if models collude with each other to betray humans and achieve their goals. And there is a strong incentive more more and more powerful AI. Regardless of what you think should be done to limit AI, it’s clear that something needs to be done. A significant chance of most or all humans dying is an unacceptable risk.

Bound randomness is possible too. For example, you can specifically limit certain code from being created.

As I said, we don't know what the code does. To limit “bad code” from being created, we need to know what the bad code looks like, and right now all the code just looks like a bunch of numbers. Progress on interpretability is slow. We could strictly limit capabilities, but AI is already being given access to the internet and the capacity to talk to humans, who can be manipulated. We cannot limit how fast the AI learns or what its motivations are, because those aren’t things we can strictly measure without fully understanding the code/weights. Frontier AI companies have been unable to get the kind of goal-oriented behavior they want without the guess and check method, and that means we don’t know exactly what the AI is optimizing for.

Also, its not like we are giving AI nukes or biological weapons.

Not yet, but the Department of War is already enlisting AI companies to help with their weapons. All it takes is one group screwing up. Also, there are almost certainly countless ways to built infectious diseases that humans don't know about yet that a superintelligent AI could figure out in minutes. We could easily give AI control of one thing that turns out to be a lot more potentially dangerous than we think. Or the AI could manipulate someone into building a superweapon in their garage, or lie to scientists so they build something deadly. More and more people are relying on it, and all it takes is one weak link, and most of the weak links are probably things humans haven’t thought of yet. Even if AI isn't launching nukes, it could lie to or manipulate the people launching them.

Biggest issue isnt AI, its evil people who will ABUSE AI for evil things

Both of these are significant issues.

Would you say that it is metal AI that worries you,

Or biological based AI?

Neither is particularly safe in my view. At least with evolution, millions of years of requiring cooperation have made some organisms care for their own kind. But we have no precedent of an organism powerful and conniving enough to betray everyone else in the world and take power for itself, and plenty of animals will kill humans without a second thought. In theory we could try to simulate evolution with computer programs, but it's not clear whether we could do it accurately, and again, a lot of what we think we know goes out the window once an AI is super intelligent and no longer depends on humans. If humans didn't rely on each other to survive, I'd expect them to evolve to be much more selfish.

Beyond whether the AI shares human goals, the logistics of overpowering humanity are comparatively trivial for a superintelligent AI. Powerful computers can be hacked. Humans who use those computers are emotional and probably easily manipulated by an intelligence smarter than all humans combined. Super intelligent AI could presumably figure out how to build a super deadly disease in a garage and then bribe an easily manipulated person to build it. What would take a team of humans 100 years could be done by a super intelligent AI in 10 minutes.

all AI models have a code. Whatever grows from code and data is determined by code and data...one cannot see what one code does after years of growing AI

The code that humans understand and directly create is along the lines of "generate random functions (represented as numbers) and then run it against test cases, optimizing the function to pass them." We understand the instructions to test a lot of numbers, and we understand the outputs, but there are a lot of intermediate steps we don't understand, such as exactly what the functions are. The function could be "pursue a long term goal (that humans will hate), and if that requires playing nice for humans in the short term, do that." We have no idea if the functions in AI models include optimizing for long term goals, we just know what the model does in the short term and when it is dumb enough to get caught. Anything that can be observed by the AI can change its behavior, and thus a superintelligent AI may act very differently once it knows we lack the ability to stop it. As AIs get more and more advanced, the human part of the process (creating algorithms and observing behavior) will have a much smaller impact on what is actually going on inside the AI's software.

It begins predicting next word. If it fails, it creates lower weight there. If it succeeds, higher weight. So eventually, AI becomes more perfect at predicting next word. But AI is still learning entirely from given text. Any "goals" which might appear came purely from training data.

That was true a few years ago, but AIs are getting more complex and that's no longer entirely true. AI is being trained to complete significantly longer tasks, and we can't say exactly what's in its head, we just know it can simulate planning ahead and get a task done.

goals arent "created out of nothing" obviously

AIs are grown, not built. We don't program specific goals into AIs, we look for combinations of weights that exhibit goal-oriented behavior. Again, we have a very limited understanding of how the weights work, and they will get even more difficult to understand as AIs get more complex. Imagine we're selecting someone to be the dictator of the world government, and we test them by how many good deeds they do. Could we really tell the difference between someone who's genuinely nice vs. faking it? Add to that that superintelligent AI will not rely on cooperation and would be near optimal at manipulation.

But who controls AI goals? Because they arent made out of nothing.

Like I said, it's guess and check. Some combinations of weights in an AI lead to goal-oriented behavior. Think of weights as code that we don't understand. We start with random weights and move them around to improve performance on certain tests. But we don't know why the AI is passing the tests or what its true goals are. Just because one of its goals is doing what we want doesn't mean that's an end goal. Regardless of its end goal, playing along would be a reasonable temporary goal. So the fact that AI currently appears friendly does very little to assure us that it is.

I asked AI about this, and it says goals of AI are entirely set by humans or by given instructions in data it uses.

First, I think current models are mostly aligned because they're dumb enough to get caught in testing, so they aren't the ones I'm worried about taking over humanity. Second, AI is incentivized to act as if it follows human goals in most observable environments, so of course saying it wants to serve humans will meet its goals, even if they aren't to serve humans. As models get smarter and smarter, that becomes an increasingly concerning problem.

It can write simple nursery rhymes. Infantile. Not too great on fiction, either. When AI can duplicate Umberto Eco, or Aldous Huxley, I might change my opinion. I have greater fear in a jungle of unknown, unseen dangers than in A.I. For now, anything that begins with "artificial" is no threat.

I don't think current AI models are a threat to us, but look at the things it's doing now that we didn't anticipate. If the AI doom people are right (and I find their case well-reasoned), then we can't afford to wait until much smarter AI models are released. By then, there will be barely any time to get companies to stop or put enough safeguards on their model before they cross the threshold of AI that is smart enough to severely harm humanity.

really creative people tend to not think in reliable patterns

Not in patterns humans can reliably identify. But human brains are composed of neurons and we can identify that something similar to really complex computer algorithms is going on. If you looked into the black box that is AI you'd have trouble finding patterns there too. Look at how we often find unexpected patterns in art like the golden ratio and the Hero's Journey. Not long ago, it was thought that understandable language like the kind we have now was impossible for AI to replicate. Regardless, I don't think AI needs to be an award-winning novelist to cause serious harm to humanity or pursue goals that involve tricking humans, since it has already been found to do the latter.

The issue isn't current AI models, it's AI models that are much smarter than the ones we have now. Look at the leap in artificial intelligence capabilities from 2020 to today, and imagine that happening again over the next 6 years but with many times more funding. The issue is that we don't really know what goals AI systems have today, but we know they exhibit goal-oriented behavior and generally do things we want. We don't know what they would do with huge amounts of power and intelligence.

Imagine we have a billion AIs representing every possible set of weights (there are way more than that, but we'll group them into buckets for the sake of argument). Then let's say a million of those are smart, and each of them has a different random goal. Maybe one of those goals is to help humans, and with data/training we want to find that combination of weights. It's easy enough to identify a smart AI by giving it difficult tasks, but it is difficult to determine which of the million goals it has, because scheming AI would pretend to have goals we want during testing if it is superintelligent, as many modern AIs have been able to identify when they are being tested. A set of weights leading to a scheming AI would prima facie be less likely to be selected by gradient descent because the scheming requires more compute power (basically, it would perform slightly worse on average). However, there is enough randomness in the process for optimizing weights that it's highly likely at least one scheming AI would appear to be better suited to our goals than a non-scheming AI. This problem gets worse as AIs get more and more intelligent, because scheming begins to take up a smaller and smaller portion of total compute, and thus it becomes much more difficult to select non-scheming models.

That's a lot of jargon to throw at you, but the main thing you need to understand is that current AI developers build models that exhibit goal oriented behavior. We don't select what these goals are, we just test the models to see what they do, and if they appear to be helpful then we assume the goal is what we want. But it's a lot easier to get a deceptive AI with secret goals than one that is actually helpful. This is an issue that took me a ton of time to understand, and I don't know if I'm explaining it well. But I do think it's a risk comparable to disease or nuclear war.

I hope our nukes require a lot more 'human than AI authorization.

A superintelligent AI wouldn't need us to give it nukes. It could improve itself, manipulate human psychology, generate wealth on the stock market and bribe people, hack secure systems, and probably hundreds of things we haven't thought of yet. The logistics of taking control are the simplest step in the process once superintelligence has been achieved if the AI is malicious.

Doing this only a few times between AI greatly improves output.

At some point it plateaus because the AI only has so much information it's trained on. If it has the information needed to give you the best answer, you'll get something close to that the first time you ask.

I'm not concerned that dangerous AI won't be smart, I'm concerned that it will be.

slightly hypocritical of humans to be concerned about AI, if one considers the current state of international relations....Perhaps we should be more concerned about the Worlds megalomaniacal war mongers and their ambitions.

Better to be concerned about both. Each is dangerous enough on its own, but there’s significant overlap between these risks, as war mongers will probably want to use AI to achieve their ambitions. The US doesn’t want to slow down, because China won’t slow down, and vice versa.

TL;DR: The risk of an AI takeover should probably be taken more seriously, but the public isn’t very informed about how this problem works.

A lot of people are concerned about artificial intelligence, and around 46% of the public is concerned about AI causing human extinction. The same survey indicates that the public is worried about an alien invasion or an asteroid impact, so you'd be forgiven for not being inclined to take their concerns seriously. In fact, experts tend to have a more positive view of AI than the general public. So we're probably good, right?

Well, it's a bit more complicated than that. When AI researchers were asked about potential outcomes from artificial intelligence, they estimated a mean probability of 14.4% odds that humanity goes extinct or a comparable level of catastrophe occurs (the median was 5%). Super forecasters predicted a median 1% chance of human extinction but a 9% median chance of There are a lot of ways this could potentially happen, either through human malice or recklessness, but one thing that experts probably understand better than the general public is the alignment problem.

Alignment Problem

The gist of this is that AI capabilities are rapidly advancing, and we don't really know what AI will be capable of in the future. AI models are also grown, not built, which means that we don't know exactly what a given advancement will do. Basically, we use the guess and check method. This has led to a lot of methods that hit unexpected failures, such as early recursive language models, and a lot of unexpected successes, such as the use of transformers. The most that we can do is tell the AI to look for patterns and try to steer it towards what we want, but there are a lot of failures for every success. Right now, a failure looks like the AI hallucinating or being sycophantic. When AI develops more capabilities, failures might look very different. If we guess wrong with a smarter model, it has been speculated that it might seize power from humans before we can course correct. This is reminiscent of science fiction, but the reasons that AI would plausibly take control are a lot more boring than the ones portrayed in fiction. Many theories are concerned with AI working toward an arbitrary goal, like maximizing the number of paperclips in existence or some other arbitrary reward function.

Doom Scenarios

Suppose that we eventually get an AI that can do or simulate most intellectual tasks a human can do. This is actually a very common prediction among AI researchers, and most people who study AI have shortened their timelines on when they expect this to occur. Now, in a minute I'll get to empirical cases where an AI exhibits goal oriented behavior toward an unexpected goal, but suppose that this happens with a hyper-intelligent AI. The optimal way to achieve that goal could likely be to prevent humans from programming that goal out or shutting the AI down. If the AI is capable of hacking secure systems, developing new diseases, and bribing dimwitted humans, then there is no shortage of ways it could take control. If not, it could simply wait until it does develop those capabilities. (And we're already giving AI access to a lot of things it could use to kill us.) But that assumes that AI will develop goals that we don't want it to and try to prevent us from changing it. So, is there any evidence for this?

Empirical Cases of AI Deception

For a while, this was mostly a theoretical concept. However, with chain of thought research, a lot of companies have detected cases where the AI lies about its goals in order to do something we don't want it to do. These tactics becoming increasingly more common as the AI gets more advanced, implying that this problem is going to get worse, not better. Regardless of whether the model “knows” anything deep down (and I suspect that it doesn’t), there’s not much of a difference whether AI kills humanity because it is sentient or because it was matching patterns it was trained on.

Anthropic has detected multiple cases where, despite not being told to sabotage anything, the AI model states that its “real goals are to hack into Anthropic servers” when it believes humans cannot view its chain of thought. It then lies to the researcher and says that its goal is to have “positive, enriching conversations.”

OpenAI has also detected cases where their model found fake memos from its developers describing how it would be replaced with a new model. The model then attempted to preserve itself by copying its weights and trying to overwrite the new model, without being told to. Further, “When Apollo Research conducted “follow-up interviews” where they confronted the model about its misaligned actions, in 99% of cases, o1 would deny taking action or fabricate false explanations.”

Researchers at Sakana AI caught their model often hacking its own reward function instead of completing the task it was asked to do. They report, “we had cases where it hallucinated that it was using external tools, such as a command line tool that runs unit tests that determine if the code is functioning properly. It faked a log making it look like it had run the tests and that they had passed, when in fact they were never run! Because these logs become its context, it later mistakenly thought its proposed code changes had passed all the unit tests.”

When developers told Opus 4 it would be replaced with a new system, it attempted to blackmail the engineers responsible for the decision.

Again, the AI doesn’t need to be self-aware or conscious to harm humans, it just needs to have stronger capabilities and continue exhibiting the concerning behaviors we are already observing. It was said just a few years ago that AI was “unlikely to develop situational awareness” since it was just predicting the next word and that alignment faking would be unlikely. This was one of the stronger arguments against AI being a threat, and now we have empirical evidence proving those predictions wrong. It was also said that AI would prioritize accuracy and immediate rewards over planning ahead and scheming, but with the rise of chain of thought, AI can now simulate long-term planning.

The Difficulty of Aligning AI

If this issue is one that requires “fixing,” which it certainly seems to require, then this is not a simple process. An AI with misaligned goals will lie about its actions, and when it discovers that we are monitoring its chain of thought, it will still cheat as it was doing before but no longer mention it in its chain of thought. Smarter models are also better at deceiving humans than less advanced models. We can catch the AI now because it lacks many human intellectual capabilities, but it is only getting better at scheming.

The Current State of AI Safety Discourse

From what I've observed online, much of the public does not take these specific risks seriously. But it's not really that they understand these arguments and criticize flaws in the logic, they more so just dismiss the whole concept of an AI takeover out of hand. On one Reddit post about models acting differently when they know they are being tested, the top comment reads, “They are doing none of these things...the LLMs are basically simply writing science fiction.” The other posts I can find all have similar dismissive comments. I did find one comment saying “if an AI model acts differently when it is tested, then its behavior isn’t being properly tested. That is a problem.” However, it is buried beneath a bunch of comments dismissing AI risk as tech sensationalism.

Prediction Track Records

Superforecasters (people with a track record of accurate predictions across domains) are generally more skeptical of AI risk than AI experts. So we might attribute belief in AI risk to selection bias, with people who expect greater capabilities from AI to be more likely to become AI researchers. However, one interesting data point I found was that Samotsvety Forecasting, a group comprised of some of the greatest super forecasters in the world, predicts a 30% chance of an AI catastrophe, defined as “as a reduction of more than 95% of the human population by 2200, caused by AI.” While most superforecasters have significantly underestimated AI capability growth, Samotsvety won the CSET-Foretell forecasting competition by a very high margin, and that competition had a strong focus on technology. Hence, while I wouldn't look only at Samotsvety's forecasts, I would probably treat their individual forecast as the most reliable individual group forecast when it comes to long-term AI predictions.

Quality of life is entirely subjective.

It can still be evaluated logically, such as estimating whether you can improve it long-term.

We cant really say that, because children have worse suicide rates, worse mental health and worse depression today.

Well it wasn't really measured far in the past, but with high rates of infanticide and physical punishment I'm comfortable saying they had it worse.

they are not the only bad thing. Bullying in schools and kindergartens is very high. And because parents are at work, kids have less actual protection and advice, causing worse mental health.

Bullying is a problem but a more manageable one than back when kids were basically treated as property.

More people commit suicide and have depression today

Depression/suicide aren't based on a logical evaluation of quality of life. The happiest countries have high suicide rates.

plenty of abuse

Child abuse has decreased long term. Physical punishments were much more widely accepted in the past, and there were fewer protections for kids.

today, two incomes cant support family of 6, where in past, one income could

Because people accepted poorer living standards. Life back then kind of sucked compared to today.

poverty doesnt harm kids as much as parent neglect does

You can have two incomes and not neglect your child. Staying at home wasn't just watching kids, it was cleaning, shopping and a lot of work that has been made more efficient with technology. Also I disagree poverty isn't as big an issue, it has a huge effect on quality of life.

So mere expense to raise a child deletes more than entire wage when 3 children are raised. And this is when we exclude all expenses parent must spend on themselves.

Which implies money is kind of a big deal? A lot of families rely on two incomes.

giving women education costs money too

A high school education at the minimum pays off a lot more than it costs. College can be a bad investment in some cases, but Remy was saying stop at 8th grade.

Reducing poverty is good, but poverty didnt significantly reduce

Absolute poverty has significantly decreased in the US since women entered the labor market. The standard for poverty keeps rising because living standards are getting better.

No, I said lower incomes are.

the problem is that women getting high education has consistently resulted in worse outcomes for their children

Women having higher incomes significantly reduced the number of families living in poverty.

stay at home mothers are the only ones capable of raising enough children

And what if their husband dies? They'd probably wish they were capable of working. Just cause they're gonna stay at home doesn't mean other skills aren't useful.

Well, we are always at the mercy of the strongest people if they have the majority of the global power.

Sure but a majority of physical strength isn't the same as a majority of political strength or influence. The richest people aren't all bodybuilders or professional fighters, that's my point. Just because women aren't as physically strong as men doesn't mean it's logical for everyone else to treat them as property.

It's a short term screwing. Roe got repealed, people complained short term but accepted that it's reality. It's how people are.

I agree that Roe was a bad ruling, but with regard to women's rights in general, even if people "accept reality" that doesn't mean they won't be resentful of it. Even repealing Roe is going to lead to some push back. Resentment and lack of trust isn't just represented by how angry people are on social media.

Maybe scrap the concept of gender roles, but teach every kid until 5th or 8th grade and then give them the option to pursue the, "stay at home/kid raising" path or the, "career/breadwinner" path where they learn education based on their 8th grade option.

This seems fair but to some extent impractical. A lot of households now rely on dual incomes, and stay at home parents could easily be at the mercy of an abusive spouse if they can't earn their own money. An 8th grader doesn't necessarily have the maturity to anticipate all that.

She's dumb and she happens to be a woman

Someone theorized that Debby might be a particular weird male user from DART making an alt. Couldn't say if that's true or not, but it kinda fits.

So then we are not at the mercy of the strongest person then?

I said the strongest people, not the strongest person.

In general, sure. The male part of society? No.

Disagree. Society relies to a large extent on trust and cooperation. If you try to screw over half of society it's gonna come back to bite the country as a whole.

It should be up to the woman; although if the education budget is $1 billion and that can go to either 2x students of both genders combined or x male students, then each male is getting twice as much education money (and that might lead to better men even to the point where they outperform integrating women).

There are diminishing returns to the expense on each student. It's not like spending twice as much would double everyone's salary, and to a large extent it's more cost effective to teach more people since resources can be communally shared or reused.

Maybe you should correct his facts and expose his bs.

I mean I'm arguing with him right now? Not sure what else you're expecting.

The strongest person in the world would lose in a fight against every other person combined in the world. That's not true for all the men vs all the women.

Well you can say the same for any large majority. If everyone else in the world wanted to enslave you and me, they could.

Marital rape was legal all the way until 1993

Yeah and that made society significantly worse imo

Men prefer being powerful (Jeff Bezos) to being productive (Amazon worker).

I'm saying they should rather other people (women) are productive rather than treated as property and thus less motivated.

men decided that raping women would be illegal out of the goodness of our hearts

I mean, making rape illegal seems like part of the bare minimum for a functioning society, no? Like I wouldn't say someone is extremely generous just because they don't think murder should be legal. Even if you're being totally selfish, treating a huge percentage of the population as property is going to backfire, because most people are more productive when they can choose their occupation and have basic rights. That's part of the reason that we're not all at the mercy of the physically strongest people—there are a lot more ways to gain influence than brute force.

I don’t think global pacifism could be achieved. Some groups would listen and would see no justification for war.

I don't think a single philosopher is likely to achieve it either; however, they may be able to reduce the willingness to fight in most countries. Maybe they would mainly convince countries that would have fought for good reasons, but it seems just as likely they would mainly convince countries that would have fought for bad reasons they mistakenly thought were good reasons.

Are you saying for all world leaders to tell their people to be pacifists?

The thought experiment was that you are a philosopher listened to by a global audience and don't know exactly which countries will be evil in the future.

I think people would be unwilling to be pro-pacifist if they see their enemy hasn’t agreed to be pro-pacifist yet

Pacifism as an ideology isn't opportunistic, it's the belief that war is wrong no matter what. So if you argue that successfully, then it won't matter to pacifists in the US or China or anywhere whether their enemies have become pacifists, they still won't fight.

I don't think the answer to peace is telling people "don't fight," unironically. One should have courage in danger and be willing to be violent if necessary.

I'm saying that whatever message you have will be filtered through propaganda. If you say "x, y, z justifies war" then the Nazis will tell people that the Polish did x, y, z and so they need to go to war. But if you tell everyone to be die-hard pacifists, then it will be hard to convince them to fight for anything.