Savant

Doing this only a few times between AI greatly improves output.

At some point it plateaus because the AI only has so much information it's trained on. If it has the information needed to give you the best answer, you'll get something close to that the first time you ask.

I'm not concerned that dangerous AI won't be smart, I'm concerned that it will be.

slightly hypocritical of humans to be concerned about AI, if one considers the current state of international relations....Perhaps we should be more concerned about the Worlds megalomaniacal war mongers and their ambitions.

Better to be concerned about both. Each is dangerous enough on its own, but there’s significant overlap between these risks, as war mongers will probably want to use AI to achieve their ambitions. The US doesn’t want to slow down, because China won’t slow down, and vice versa.

TL;DR: The risk of an AI takeover should probably be taken more seriously, but the public isn’t very informed about how this problem works.

A lot of people are concerned about artificial intelligence, and around 46% of the public is concerned about AI causing human extinction. The same survey indicates that the public is worried about an alien invasion or an asteroid impact, so you'd be forgiven for not being inclined to take their concerns seriously. In fact, experts tend to have a more positive view of AI than the general public. So we're probably good, right?

Well, it's a bit more complicated than that. When AI researchers were asked about potential outcomes from artificial intelligence, they estimated a mean probability of 14.4% odds that humanity goes extinct or a comparable level of catastrophe occurs (the median was 5%). Super forecasters predicted a median 1% chance of human extinction but a 9% median chance of There are a lot of ways this could potentially happen, either through human malice or recklessness, but one thing that experts probably understand better than the general public is the alignment problem.

Alignment Problem

The gist of this is that AI capabilities are rapidly advancing, and we don't really know what AI will be capable of in the future. AI models are also grown, not built, which means that we don't know exactly what a given advancement will do. Basically, we use the guess and check method. This has led to a lot of methods that hit unexpected failures, such as early recursive language models, and a lot of unexpected successes, such as the use of transformers. The most that we can do is tell the AI to look for patterns and try to steer it towards what we want, but there are a lot of failures for every success. Right now, a failure looks like the AI hallucinating or being sycophantic. When AI develops more capabilities, failures might look very different. If we guess wrong with a smarter model, it has been speculated that it might seize power from humans before we can course correct. This is reminiscent of science fiction, but the reasons that AI would plausibly take control are a lot more boring than the ones portrayed in fiction. Many theories are concerned with AI working toward an arbitrary goal, like maximizing the number of paperclips in existence or some other arbitrary reward function.

Doom Scenarios

Suppose that we eventually get an AI that can do or simulate most intellectual tasks a human can do. This is actually a very common prediction among AI researchers, and most people who study AI have shortened their timelines on when they expect this to occur. Now, in a minute I'll get to empirical cases where an AI exhibits goal oriented behavior toward an unexpected goal, but suppose that this happens with a hyper-intelligent AI. The optimal way to achieve that goal could likely be to prevent humans from programming that goal out or shutting the AI down. If the AI is capable of hacking secure systems, developing new diseases, and bribing dimwitted humans, then there is no shortage of ways it could take control. If not, it could simply wait until it does develop those capabilities. (And we're already giving AI access to a lot of things it could use to kill us.) But that assumes that AI will develop goals that we don't want it to and try to prevent us from changing it. So, is there any evidence for this?

Empirical Cases of AI Deception

For a while, this was mostly a theoretical concept. However, with chain of thought research, a lot of companies have detected cases where the AI lies about its goals in order to do something we don't want it to do. These tactics becoming increasingly more common as the AI gets more advanced, implying that this problem is going to get worse, not better. Regardless of whether the model “knows” anything deep down (and I suspect that it doesn’t), there’s not much of a difference whether AI kills humanity because it is sentient or because it was matching patterns it was trained on.

Anthropic has detected multiple cases where, despite not being told to sabotage anything, the AI model states that its “real goals are to hack into Anthropic servers” when it believes humans cannot view its chain of thought. It then lies to the researcher and says that its goal is to have “positive, enriching conversations.”

OpenAI has also detected cases where their model found fake memos from its developers describing how it would be replaced with a new model. The model then attempted to preserve itself by copying its weights and trying to overwrite the new model, without being told to. Further, “When Apollo Research conducted “follow-up interviews” where they confronted the model about its misaligned actions, in 99% of cases, o1 would deny taking action or fabricate false explanations.”

Researchers at Sakana AI caught their model often hacking its own reward function instead of completing the task it was asked to do. They report, “we had cases where it hallucinated that it was using external tools, such as a command line tool that runs unit tests that determine if the code is functioning properly. It faked a log making it look like it had run the tests and that they had passed, when in fact they were never run! Because these logs become its context, it later mistakenly thought its proposed code changes had passed all the unit tests.”

When developers told Opus 4 it would be replaced with a new system, it attempted to blackmail the engineers responsible for the decision.

Again, the AI doesn’t need to be self-aware or conscious to harm humans, it just needs to have stronger capabilities and continue exhibiting the concerning behaviors we are already observing. It was said just a few years ago that AI was “unlikely to develop situational awareness” since it was just predicting the next word and that alignment faking would be unlikely. This was one of the stronger arguments against AI being a threat, and now we have empirical evidence proving those predictions wrong. It was also said that AI would prioritize accuracy and immediate rewards over planning ahead and scheming, but with the rise of chain of thought, AI can now simulate long-term planning.

The Difficulty of Aligning AI

If this issue is one that requires “fixing,” which it certainly seems to require, then this is not a simple process. An AI with misaligned goals will lie about its actions, and when it discovers that we are monitoring its chain of thought, it will still cheat as it was doing before but no longer mention it in its chain of thought. Smarter models are also better at deceiving humans than less advanced models. We can catch the AI now because it lacks many human intellectual capabilities, but it is only getting better at scheming.

The Current State of AI Safety Discourse

From what I've observed online, much of the public does not take these specific risks seriously. But it's not really that they understand these arguments and criticize flaws in the logic, they more so just dismiss the whole concept of an AI takeover out of hand. On one Reddit post about models acting differently when they know they are being tested, the top comment reads, “They are doing none of these things...the LLMs are basically simply writing science fiction.” The other posts I can find all have similar dismissive comments. I did find one comment saying “if an AI model acts differently when it is tested, then its behavior isn’t being properly tested. That is a problem.” However, it is buried beneath a bunch of comments dismissing AI risk as tech sensationalism.

Prediction Track Records

Superforecasters (people with a track record of accurate predictions across domains) are generally more skeptical of AI risk than AI experts. So we might attribute belief in AI risk to selection bias, with people who expect greater capabilities from AI to be more likely to become AI researchers. However, one interesting data point I found was that Samotsvety Forecasting, a group comprised of some of the greatest super forecasters in the world, predicts a 30% chance of an AI catastrophe, defined as “as a reduction of more than 95% of the human population by 2200, caused by AI.” While most superforecasters have significantly underestimated AI capability growth, Samotsvety won the CSET-Foretell forecasting competition by a very high margin, and that competition had a strong focus on technology. Hence, while I wouldn't look only at Samotsvety's forecasts, I would probably treat their individual forecast as the most reliable individual group forecast when it comes to long-term AI predictions.

Quality of life is entirely subjective.

It can still be evaluated logically, such as estimating whether you can improve it long-term.

We cant really say that, because children have worse suicide rates, worse mental health and worse depression today.

Well it wasn't really measured far in the past, but with high rates of infanticide and physical punishment I'm comfortable saying they had it worse.

they are not the only bad thing. Bullying in schools and kindergartens is very high. And because parents are at work, kids have less actual protection and advice, causing worse mental health.

Bullying is a problem but a more manageable one than back when kids were basically treated as property.

More people commit suicide and have depression today

Depression/suicide aren't based on a logical evaluation of quality of life. The happiest countries have high suicide rates.

plenty of abuse

Child abuse has decreased long term. Physical punishments were much more widely accepted in the past, and there were fewer protections for kids.

today, two incomes cant support family of 6, where in past, one income could

Because people accepted poorer living standards. Life back then kind of sucked compared to today.

poverty doesnt harm kids as much as parent neglect does

You can have two incomes and not neglect your child. Staying at home wasn't just watching kids, it was cleaning, shopping and a lot of work that has been made more efficient with technology. Also I disagree poverty isn't as big an issue, it has a huge effect on quality of life.

So mere expense to raise a child deletes more than entire wage when 3 children are raised. And this is when we exclude all expenses parent must spend on themselves.

Which implies money is kind of a big deal? A lot of families rely on two incomes.

giving women education costs money too

A high school education at the minimum pays off a lot more than it costs. College can be a bad investment in some cases, but Remy was saying stop at 8th grade.

Reducing poverty is good, but poverty didnt significantly reduce

Absolute poverty has significantly decreased in the US since women entered the labor market. The standard for poverty keeps rising because living standards are getting better.

No, I said lower incomes are.

the problem is that women getting high education has consistently resulted in worse outcomes for their children

Women having higher incomes significantly reduced the number of families living in poverty.

stay at home mothers are the only ones capable of raising enough children

And what if their husband dies? They'd probably wish they were capable of working. Just cause they're gonna stay at home doesn't mean other skills aren't useful.

Well, we are always at the mercy of the strongest people if they have the majority of the global power.

Sure but a majority of physical strength isn't the same as a majority of political strength or influence. The richest people aren't all bodybuilders or professional fighters, that's my point. Just because women aren't as physically strong as men doesn't mean it's logical for everyone else to treat them as property.

It's a short term screwing. Roe got repealed, people complained short term but accepted that it's reality. It's how people are.

I agree that Roe was a bad ruling, but with regard to women's rights in general, even if people "accept reality" that doesn't mean they won't be resentful of it. Even repealing Roe is going to lead to some push back. Resentment and lack of trust isn't just represented by how angry people are on social media.

Maybe scrap the concept of gender roles, but teach every kid until 5th or 8th grade and then give them the option to pursue the, "stay at home/kid raising" path or the, "career/breadwinner" path where they learn education based on their 8th grade option.

This seems fair but to some extent impractical. A lot of households now rely on dual incomes, and stay at home parents could easily be at the mercy of an abusive spouse if they can't earn their own money. An 8th grader doesn't necessarily have the maturity to anticipate all that.

She's dumb and she happens to be a woman

Someone theorized that Debby might be a particular weird male user from DART making an alt. Couldn't say if that's true or not, but it kinda fits.

So then we are not at the mercy of the strongest person then?

I said the strongest people, not the strongest person.

In general, sure. The male part of society? No.

Disagree. Society relies to a large extent on trust and cooperation. If you try to screw over half of society it's gonna come back to bite the country as a whole.

It should be up to the woman; although if the education budget is $1 billion and that can go to either 2x students of both genders combined or x male students, then each male is getting twice as much education money (and that might lead to better men even to the point where they outperform integrating women).

There are diminishing returns to the expense on each student. It's not like spending twice as much would double everyone's salary, and to a large extent it's more cost effective to teach more people since resources can be communally shared or reused.

Maybe you should correct his facts and expose his bs.

I mean I'm arguing with him right now? Not sure what else you're expecting.

The strongest person in the world would lose in a fight against every other person combined in the world. That's not true for all the men vs all the women.

Well you can say the same for any large majority. If everyone else in the world wanted to enslave you and me, they could.

Marital rape was legal all the way until 1993

Yeah and that made society significantly worse imo

Men prefer being powerful (Jeff Bezos) to being productive (Amazon worker).

I'm saying they should rather other people (women) are productive rather than treated as property and thus less motivated.

men decided that raping women would be illegal out of the goodness of our hearts

I mean, making rape illegal seems like part of the bare minimum for a functioning society, no? Like I wouldn't say someone is extremely generous just because they don't think murder should be legal. Even if you're being totally selfish, treating a huge percentage of the population as property is going to backfire, because most people are more productive when they can choose their occupation and have basic rights. That's part of the reason that we're not all at the mercy of the physically strongest people—there are a lot more ways to gain influence than brute force.

I don’t think global pacifism could be achieved. Some groups would listen and would see no justification for war.

I don't think a single philosopher is likely to achieve it either; however, they may be able to reduce the willingness to fight in most countries. Maybe they would mainly convince countries that would have fought for good reasons, but it seems just as likely they would mainly convince countries that would have fought for bad reasons they mistakenly thought were good reasons.

Are you saying for all world leaders to tell their people to be pacifists?

The thought experiment was that you are a philosopher listened to by a global audience and don't know exactly which countries will be evil in the future.

I think people would be unwilling to be pro-pacifist if they see their enemy hasn’t agreed to be pro-pacifist yet

Pacifism as an ideology isn't opportunistic, it's the belief that war is wrong no matter what. So if you argue that successfully, then it won't matter to pacifists in the US or China or anywhere whether their enemies have become pacifists, they still won't fight.

I don't think the answer to peace is telling people "don't fight," unironically. One should have courage in danger and be willing to be violent if necessary.

I'm saying that whatever message you have will be filtered through propaganda. If you say "x, y, z justifies war" then the Nazis will tell people that the Polish did x, y, z and so they need to go to war. But if you tell everyone to be die-hard pacifists, then it will be hard to convince them to fight for anything.

There will always be an opposing force so long as there is evil in the world

Sure, and there will always be non-evil people who fight to stop them. But if you can reduce both factions by reducing fighting in general, you should, as it will mitigate the fallout.

I think we've seen a small increase on the site but just from old DDO and DART members. To my knowledge, no one from the YouTube channel or even Google search.

It's the end of World War I, and you're a world famous philosopher. Several armchair commentators have theorized that another world war may break out in a few decades, but they're not sure over what. Meanwhile, there is vigorous debate going on about just war theory, whether nations are ever justified in killing civilians, the non-aggression principle, and so forth. Most philosophers argue that war is justified under certain conditions but not under others. You are inclined to agree with them, although you disagree slightly on what the conditions are. So, what should you do?

Now, suppose you argue that war is sometimes justified and give a list of criteria, such as responding to aggression. When the Nazis take power, they will inevitably argue, as best they can, that they meet the criteria, and hence that World War II is justified. Now, world famous as you are, it will probably be difficult beforehand to anticipate exactly what the new world war will be fought over, and your criteria may be vague enough that either side can appropriate your argument. Even if you do exclude the Nazis when arguing what makes war justified, they can easily appropriate the rest of your argument and replace the things you said justify war with their own grievances. It's not difficult to take an argument saying that self-defense justifies violence and stretch it to argue that harms against the German people justify warfare. Add to this that the Nazis will inevitably lie about what's happening and invent a list of harms to justify war.

But suppose you instead argue that war is always wrong, that people should refuse to fight no matter what. If this logic becomes somewhat popular everywhere, even a little, it will make it more difficult for any aggressor to justify war. They can say that other nations are plotting against them, but if their population has a lot of die-hard pacifists, then those arguments won't do much to convince them.

Since you don't know who will listen to your arguments for or against pacifism, you should therefore advocate for violence not to be used ever, because if everyone uses violence less it will be a net gain for everyone. If the Axis powers use fewer soldiers and attack fewer civilians, and the Allies use fewer soldiers and attack fewer civilians, then the win probabilities remain roughly equal, but fewer people die.

Disclaimer: I'm actually not 100% sure if this logic holds up or not, but I thought it was worth sharing. A downside of this approach may be that lying about your conclusions leads to people accepting bad logical premises.

The working class didn’t just vote against their own interests.

Let's stop pretending that workers have "class" interests. They have personal interests. Hence why labor unions pushed for Chinese Exclusion Act when it was seen as in their interest.

They worshipped the billionaires robbing them

Almost every faction hates some rich people and loves others. Everyone knows "the rich" doesn't include whatever influencer is telling you to hate the rich.

They cheered while the rich wrote off private jets as tax deductions.

I don't think a significant number of poor people are cheering for this, presuming the jet is being used for personal use, which is what you're implying. If something is used for business, then it can be calculated as a business expense. Jets aren't special in this regard. The law probably gets abused, but I don't know who is cheering for tax abuse besides the tax abusers themselves.

I'm not saying we should just implement this immediately. I'm not sure the best way to test this, but I think it's of some utility to come up with potential optimal forms of democracy, because someday people might get a chance to experiment and discover this works better.

of something doesn't work on the macro we shouldn't test it on the micro or visa versa

Why not? Representative democracy is effectively testing something on the micro (direct voting on every policy) that didn't work on the macro.

In a legislature all the votes are public and you're counting hundreds of votes, not millions. It would be extremely hard to get the math wrong if it's all public and literally anyone can fact check. I'm not proposing the general public rank candidates, just Parliament.

I agree but he did mention poverty rates and median incomes

Every time RCV is tried in the real world it has more unfair practices than other forms of voting that preceded it. Terrible ideal

The criticism I've seen is that it's hard to understand, but in my system it's the legislature doing it not the public.