The Current State of AI Safety

For them, something can't happen until it 'has happened once at some point.

Correct, in so far as with plumbing, electrical construction all engineering, new codes often occur after incident occurs that harms, causes damage, etc. Ex adding in ground { green or bare } wire to the hot lead { black } and neutral wire { white } return to so breakers circuit breaker would blow before killing a person or overheating an apllicance etc.

As ive mention and posted about elsewhere around here, two seperate vids that presented info by smart people who beginning to understand how complex AI is coming to there conclusions --often good stuff in various sciences-- and it is about a geometrical stacking of 2D planes, each is considered the next higher dimenison.

These stacks may go into the millions, the the AI looks for relevantly significant lines of relationship { connections } between the 2D planes with info data bits on them, and allows AI to come to an conclusion/resultant. Max Tegmark is one of the vids. Here is the other one that I think was more complex than Max and his students. I dunno.

https://www.youtube.com/watch?v=D8GOeCFFby4

Haven’t you heard of “Antropic” AI defense contractor warning AI can be abused if it in the wrong hands and limits have to be imposed.

I understand your point, but just to be clear once more,

AI doesnt generate patterns bad for humans by default.

It generates random patterns at start. these patterns are essentially a mess at start and untrained AI is entirely incoherent.

What humans do is create algorithm in AI which converts data into useful patterns (this is AI training. Basically AI takes data and converts it to patterns).

And these useful patterns are prioritized over random mess.

And if the only thing AI has other than useful patterns is random mess, that mess is essentially incoherent in most cases.

But I have to agree with you on point of AI being dangerous if it falls in wrong hands.

On that point, I agree. If super AI falls in wrong hands, in hands of people or governments who will use it for bad things, it almost guarantees large scale destruction.

What I disagree about is AI being dangerous by default or in most cases.

It would be dangerous if AI knew more about Iran’s military strength than the Generals themselves or even Trump.

What I disagree about is AI being dangerous by default or in most cases.

Those randomly generated patterns aren’t dangerous by default, but to make the artificial intelligence “intelligent” you need to train it to exhibit goal-oriented behavior. So yeah if we just let it behave randomly it wouldn’t try to take over humanity, but at that point it’s not even artificial intelligence. Once it’s intelligent/trained, it has one of a billion possible goals which all appear the same to us and perform the same on tests. Most of those goals require the AI’s survival but not human survival, hence the danger.

AI cannot be programmed to prioritize human values and life over objective facts like warfare and disease control.

Those randomly generated patterns aren’t dangerous by default, but to make the artificial intelligence “intelligent” you need to train it to exhibit goal-oriented behavior

to be more precise, you need it to exhibit prompt oriented behavior.

AI is trained to follow prompt made by humans. It doesnt follow arbitrary goals, but goals set by prompt.

And I think in future, AI will be made in different way.

Even if you were to prove that current way AI is made is no good, it means nothing. AI in past 10 years has changed many methods of its creation. the AI you see today, "predicting next thing" patterns, is just how it is today, and it isnt superintelligent despite being trained after trillion parameters made from data.

In future, I assume AI will follow set based logic learning like math does instead of predicting next thing in data.

to put it simply, in future, AI wont learn from data. We will have algorithms which convert data to patterns, and then have those patterns be effectively used by AI.

this is my opinion. I dont share views that current AI making method will last much long. In fact, current AI uses a lot of resources and is terribly slow and hallucinates and even miscalculates.

Those are human traits that AI emulates like hallucinations and miscalculation.

AI is trained to follow prompt made by humans. It doesnt follow arbitrary goals, but goals set by prompt.

We can only verify that in some use cases. The code keeps getting tweaked until the AI follows prompts in the short term. But saying that is all the AI cares about is a bit like saying, “We filtered out all the politicians who were mean to us, so these politicians are guaranteed to always act in our interests forever.” We have no idea what’s in the AI model’s head or what the politician is really thinking, we just know that their current algorithms make them appear helpful.

In future, I assume AI will follow set based logic learning like math does instead of predicting next thing in data.

Those models exist, but they are much less efficient. There’s a big gap between code simple enough for us to understand and behavior intelligent enough to do anything a human can do.

the AI you see today, "predicting next thing" patterns, is just how it is today, and it isnt superintelligent despite being trained after trillion parameters made from data.

Again, we don’t directly program in the “predict next thing” behavior, we just know the weights of the current models appear to lead to that behavior. And there are other things we want AIs to do beyond predictions, like avoiding naughty words. That’s not directly programmed in, it’s guess and check, and the “check” is very imperfect.

AI,

Suddenly I think of genies in lamps,

Whether one should risk a wish, not knowing whether it 'might be an evil genie who will twist your wish.

Or whether one ought free the genie, not 'knowing their true character.

Or a Deck of Many Things.

Suddenly I think of genies in lamps,

Whether one should risk a wish, not knowing whether it 'might be an evil genie who will twist your wish.

It’s worse than that, because the AI isn’t guaranteed to stay true to our wishes at all. All we can do is measure whether a particular set of weights appears to create an AI loyal to humans, and per Goodhart’s law, we should not expect that strategy to actually produce AIs loyal to humans, particularly once they are much smarter than us. We’re much more likely to stumble upon an AI with an arbitrary goal than one loyal to humans, since both will act equally helpful past a certain level of intelligence (one due to actual loyalty, the other because it’s an effective way to attain more power until it can betray us).

Those models exist, but they are much less efficient.

I was talking of set based logic there, not existing models. Set based logic is what all math is based on, and what pure logic is based on. Sort of like, instead of using training data to produce probability sets by context, one divides all things on correct sets and works with that. AI today still uses set logic, but its statistical sets detecting which sets repeat more in training data, instead of using certainty sets like in math.

Circuit tracing is a relatively new technique that lets researchers track how an AI model builds its answers step by step – like following the wiring in a brain. It works by chaining together different components of a model. Anthropic used it to spy on Claude’s inner workings. This revealed some truly odd, sometimes inhuman ways of arriving at an answer that the bot wouldn’t even admit to using when asked.

This working backwards suggests LLMs might have more foresight than we assumed and that they don’t always just predict one word after another to form a coherent answer. All in all, these findings are a big deal – they prove we can finally see how these models operate, at least in part.

Again, we don’t directly program in the “predict next thing” behavior, we just know the weights of the current models appear to lead to that behavior.

Again, you are talking of today's models and methods as if they are permanent, when realistically they probably will be entirely different after just 5 years.

Now, here is what confuses me, because you say AI cannot be controlled.

I use AI to make another AI, it's a work in progress but I got the basic structure made by another AI.

It's entirely coded in python, and I can see full code of each file another AI gave me.

You are essentially saying that code is unknown, but from what I understand, the only unknown thing are weights because they start as random.

But weights only start random because making all weights be same doesn't work good because it can't tell which weight is wrong.

And these weights may be unknown, but they are essentially input transformers.

When you give input, weights transform input into output.

Now, your concern is, that there is a hidden weight or a set of them, which would transform input into something which destroys humans, but it only does so once certain it can succeed.

I say that is unlikely, because weights start as random mess.

It is like throwing letters around and expecting to form sentence "destroy humans".

But even less likely, because not all weights control output equally.

Weights are in fact dependent on input. Different input gets transformed by different weights.

I am still learning about AI, but to me, what you are describing only likely emerges if specifically trained for it.

Modern AI are specifically trained not to harm humans.

Their weights are specifically managed so that weight which protects humans carries highest importance and greatest impact.

And AI intelligence is entirely made out of weights which represent pattern generation. AI is designed in a similar way to human brain, with artificial neurons which learn based on weights entirely.

So, as I said before, the only real concern with super intelligence is it falling in hands that will train it for evil.

But we are still not at any super intelligence level. While almost every criminal in the world has access to AI and can use them for evil, so far none did too much harm to claim any doom of humanity.

And AI is trickle down technology, it very quickly found its way everywhere and became more affordable in improved versions.

Now, of course, highest level of AI is actually not available to masses, because that level requires supercomputer with countless parameters model trained on almost all available data in the world, which is I assume what top AI companies have and don't share with others.

My point is, we could already be having super AI, just not available to masses.

However, I argue for a position different than yours. I don't argue for restricting AI development.

Restricting AI development is self-defeating, because the only restricted ones will be the good guys. Evil leaders won't restrict their AI development, but will keep developing full speed.

Solution is full speed AI development, especially of linked AI models.

Because, once again, even if we are entering a Terminator movie scenario where if we develop technology, it possibly destroys us, and if we don't develop it, someone else will, forcing us to develop it anyway...

The solution is: know your enemy.

We must speed up research of AI, to be able to understand AI completely.

And maybe, modify random weights generation.

We know how each weight affects output individually, so maybe randomized weights at start can be scanned for potential threat, because we know how they modify input to affect output.

Also, just to confirm once more, your issue is with weights in AI model being random?

If that is whole source of problem, then AI isn't the problem because there are AI systems which don't use that at all.

RAG AI system doesn't use any weights. It merely retrieves knowledge from external file and generates output by transforming collected data slightly to make it appear more like natural language.

If your issue here is entirely focused on AI weights, that is only one of many AI systems.

Yes, weights are latest and most complex system, but actually, AI which is RAG plus rule based logic doesn't need weights at all, can still do some reasoning, and retrieves knowledge from data same as any AI.

It is like throwing letters around and expecting to form sentence "destroy humans".

But even less likely, because not all weights control output equally.

The issue is we only judge weights by the outputs they produce. However, for every set of weights that have high capabilities and set the goal “do the task humans want,” which gives us the desired output, there are far more combinations of weights with high capabilities that say “maximize x random thing,” which is accomplished best by doing what humans want right now and playing nice, then betraying humans. So we are basically tuning weights until we get a high-capability program that does what we want right now, but most high-capability programs that do we want right now would betray humans, we just don't know which are which.

It's a bit like generating random numbers until you get a number between 1-10, but 9 of the numbers between 1 and 10 kill you. You could say “most numbers won't kill us,” but most of the ones that don’t kill us also don't do the tasks we want, so we keep tweaking the weights until we get a model that does what we want, and if the model is capable of scheming, we probably get one that kills us. When models are dumber, most models with arbitrary goals aren't clever enough to trick us, so they get filtered out. When models get really smart, even the ones with goals that require betraying humans will be smart enough to play along.

RAG AI system doesn't use any weights

Typically it’s paired with an LLM that does use weights.

AI which is RAG plus rule based logic doesn't need weights at all

Rule logic needs to be programmed manually by humans and takes far longer. Good luck getting companies to prioritize safety over performance, when they’ve shown far too little interest in safety so far.

Restricting AI development is self-defeating, because the only restricted ones will be the good guys. Evil leaders won't restrict their AI development, but will keep developing full speed.

Pausing AI development globally would require international cooperation of the likes we’ve never seen before, something akin to restricting the development of nukes. I don’t know if it’s possible to ever accomplish, but I want people to know the risks, because that’s the only way humanity has a fighting chance. Eliezer Yudkowsky, who is even more pessimistic than me, stated, “The moratorium on new large training runs needs to be indefinite and worldwide. There can be no exceptions, including for governments or militaries. If the policy starts with the U.S., then China needs to see that the U.S. is not seeking an advantage but rather trying to prevent a horrifically dangerous technology which can have no true owner and which will kill everyone in the U.S. and in China and on Earth.” That’s a tall order, and I don’t think we could ever get the kind of pause that makes Eliezer happy, but with some cooperation we might be able to slow development as a whole or require more testing and red lines. Getting everyone to understand the threat that this poses is a start, and that’s what my goal is right now. Society will decide how to handle this, but I’d rather they decide on a policy from a position of knowledge of the risks rather than a position of ignorance.

The issue is we only judge weights by the outputs they produce. However, for every set of weights that have high capabilities and set the goal “do the task humans want,” which gives us the desired output, there are far more combinations of weights with high capabilities that say “maximize x random thing,” which is accomplished best by doing what humans want right now and playing nice, then betraying humans.

So your concern is essentially that weights which manage output only do so to please their goal.

Weights dont have a goal.

Each weight has its own unique value which modifies output. We can know value of each of these weights, it is just not possible to read all values because there is trillions of them.

to put it simply, AI doesnt have goal or desires. It merely has weights which transform input into output. For example, prompt "hello" gets converted into a vector number and multiplied by bunch of weights one after another, and when you have trillions of weights and also bias number, output can be non-repetitive, but it can also be trained to adjust weights so output becomes desirable.

In simple terms, AI transforms, input into output.

Unless someone changes the input, they will get somewhat similar output each time. these weights dont "think". they actually multiply input vector, which seems like thinking, but its actually just a very advanced calculator with bias added.

It is true that we dont know exactly what happens because no one can observe trillions of weights. But weights are adjusted during training. We can adjust them, tho we dont do so manually.

Rule logic needs to be programmed manually by humans and takes far longer. Good luck getting companies to prioritize safety over performance, when they’ve shown far too little interest in safety so far.

Actually, this is where I want to add something entirely new. First, RAG AI with rule based logic, and ability to deduce in form of sets, yes, it takes time to program and improve. It would actually have to have a dictionary of its own in it to be able to convert sentence to multiple sets to be able to reason well. But there are two things: We can actually see the entire code. there are no unknowns. It works just as efficiently as regular AI in retrieving information, tho slightly worse at reasoning. But its not like today's AI is much good at reasoning anyway. And second, we can have AI program it. I dont know a lot about programming, but I do use AI to program first a chatbot, then a retriever, which were successful, and then through repeated trial and error, I also made chatbot capable of next word prediction, which was good but slower than retriever. then finally, I realized retriever which combines different sentences into new answer is probably best and fastest, and by converting search sentences into tokens, search function is made much faster. And added randomness makes it non-repetitive. My next step is to program RAG AI retriever, which I already have code for, but also make it be able to: 1. Convert learned text into sets, 2. Be able to do deductive logic using those sets. For example, "cats" belongs in set of "animals", "kitten" belongs in set of "cats", the AI can deduce that kittens are animals without having that knowledge at start. (Kitten included in Cats. Cats included in animals. Kitten included in animals). this form of deductive logic works entirely by converting sentences to sets, exploring what each set contains as well. Sort of like advanced language calculator.

Pausing AI development globally would require international cooperation of the likes we’ve never seen before, something akin to restricting the development of nukes. I don’t know if it’s possible to ever accomplish

It isnt possible to accomplish, because all countries and groups wont agree. today, anyone can code an AI model using an AI itself as programmer, using repeated trial and error so AI is told where one mistake is, fixes it, then program is tested again, if mistake is fixed, it is moved to next mistake. It doesnt even take much time or programming skills anymore, just basic "test, find error, fix error, repeat" method.

So literally anyone can have a local AI, entirely programmed by themselves, trained on whatever data. Now, not everyone has a supercomputer, but those are getting cheaper too and pretty much most larger groups in world can afford them.

So trying to stop AI development would only harm good guys who want to stop it, and bad guys would keep developing and be even ahead of good guys in development.

Weights dont have a goal.

AIs can exhibit goal-oriented behavior even without having conscious desires, and some combinations of weights absolutely lead to goal-oriented behavior, such as the ones AI has now. Through RLHF, AI learns to act as if it has the goal of helping me. So either it has the terminal goal of helping me or some other terminal goal that is aided by acting helpful; regardless, AIs are perfectly capable of exhibiting goal-oriented behavior.

[RAG] works just as efficiently as regular AI in retrieving information, tho slightly worse at reasoning. But its not like today's AI is much good at reasoning anyway

Reasoning is what companies are focused on optimizing right now. They’re not about to switch to RAG without significant pressure. If you want them to switch, then great, that’s a massive safety improvement. But if we want companies to care about safety, then everyone needs to understand the significance of this problem.

And second, we can have AI program it.

That’s just kicking the can down the road, because then we need to align the AI doing the programming.

because all countries and groups wont agree

We managed to stop nuclear war and prevent some countries from getting nukes. Pausing AI will be much more difficult, but slowing it might be feasible.

literally anyone can have a local AI, entirely programmed by themselves, trained on whatever data

Not anyone can build premier AI models. Right now, it requires extremely high compute and teams of researchers. We know who is the closest to AGI right now, and forcing them to test more or even slow down a little would reduce risk.

China leads in AI. Their new year fireworks were all controlled by AI and it was a magnificent show.

some combinations of weights absolutely lead to goal-oriented behavior,

Its not "goal oriented behavior". It is AI transforming input using weights according to your goal.

Again, AI transforms input. thats what it does. the whole idea of AI is very simple: Have large combinations of weights, each transforms input vector number.

It is not "sentient" nor has any specific goals or wants.

All it does is:

1. transform input to vector number
2. Multiply vector number with weights
3. Predict probability of next token (or in images, next step)

So the evil desires (evil set of weights, to be precise) which you talk about dont need to exist in AI. they can exist, but dont really have to.

Reasoning is what companies are focused on optimizing right now. They’re not about to switch to RAG without significant pressure. If you want them to switch, then great, that’s a massive safety improvement. But if we want companies to care about safety, then everyone needs to understand the significance of this problem.

But do you agree that highest certainty of reasoning is made by deduction and sets? Companies are working hard to improve AI's reasoning, yet AI still makes bad miscalculations, hallucinations, and even blatant lies.

I think RAG or smart retriever system can actually be made with reasoning ability, but one must first convert human language into working system of sets. Deductive reasoning in math is made entirely out of sets, and same can be done with human language if words are treated as sets which contain other words. that enables precise deductive reasoning without all the AI decades of training nonsense.

That’s just kicking the can down the road, because then we need to align the AI doing the programming.

No, because we can see the code AI makes. We can see whole code for new AI made by AI. Because new AI doesnt use weights, all code is clearly visible, and harm easier to detect.

We managed to stop nuclear war and prevent some countries from getting nukes.

We didnt succeed in abolishing nukes tho. It only takes one bad government to lead world into nuclear war. But we cant give up improving nukes and delivery systems because that would only help our enemies, not us.

It is AI transforming input using weights according to your goal.

The idea is you could look for weight combinations that pursue your goal and stumble upon weight combinations pursuing another goal, that appear to pursue your goal.

No, because we can see the code AI makes. We can see whole code for new AI made by AI. Because new AI doesnt use weights, all code is clearly visible, and harm easier to detect.

I don't think it would be easy to check for hidden vulnerabilities or exploits though. Plus to get a RAG that is x intelligent, you need a black box AI that is at least x intelligent, or a lot of human effort, which will fall behind companies willing to use black boxes. The black box AI just existing is dangerous because humans can be manipulated and computers can be hacked, and the technology existing means someone else may use it much less carefully.

We didnt succeed in abolishing nukes tho

We’ve succeeded in reducing the number and adding more safeguards.

Disclaimer: I did some more digging, and I think Constitutional AI is more promising than I gave it credit for. When Claude had hidden goals, it still pursued the spirit of the constitution it was given. So deceptive alignment may not be as hard a problem as I thought, at least based empirically on recent models. I don’t know how well this will generalize to smarter models though.

The idea is you could look for weight combinations that pursue your goal and stumble upon weight combinations pursuing another goal, that appear to pursue your goal.

Yes, but I am saying extremely dangerous AI is unlikely, and more and more unlikely the more weights are adjusted for safety measures because that also changes "goal ai pursues".

I don't think it would be easy to check for hidden vulnerabilities or exploits though.

Checking code is easy, when you realize AI code isnt exactly long. its knowledge base which takes most of storage. In weights AI, weights take a lot of storage. Now, RAG AI doesnt use weights. It can still use vectors like today's AI, it can still do calculations like calculator to answer math questions, it can still retrieve knowledge effectively, and with good combination of vectors and sets logic, it could even be made able to reason in human language too in a way which is as good as current AI or even better if deduction guarantees precision.

Plus to get a RAG that is x intelligent, you need a black box AI that is at least x intelligent

Sort of. AI can make another AI. that is next step in evolution of AI. AI which reproduces itself or produces similar or even different AI. However, AI can also make AI which is better than it, in this specific case, AI with weights can make AI without weights.

the reason why I argue for working on AI without weights is because weights are unknown area, and we could really be making anything there, and also, we could be wasting a lot of resources on something which wont lead anywhere. Because, as I said before, reasoning using deduction sets is mathematically perfect. Reasoning built on training data is only as good as training data.

The black box AI just existing is dangerous because humans can be manipulated and computers can be hacked, and the technology existing means someone else may use it much less carefully.

What I simply think is that a different type of AI needs to be developed, because problem with current AI is that it is already too complicated. We have already reached a point where we cant understand it, and that is dangerous on its own.

We’ve succeeded in reducing the number and adding more safeguards.

the world is still developing nuclear weapons and means of delivery as fast as ever. What we reduced is number of old nuclear bombs. the new ICBMs are pretty much being developed on yearly basis, and those ICBMs are more dangerous than nuclear bombs themselves, because nuclear bomb is nothing without means of delivery.

But when it comes to AI, problem is, AI isnt something owned by only 9 countries's governments. AI is pretty much owned almost universally. I know a very advanced AI currently isnt owned by masses, but almost all advanced models are, and masses can even built their own AI.

And also, countries agreed to limit nuclear weapons because it was simply pointless to have more. they already have more than enough to wipe out planet 7 times.

Countries will never agree to limit AI, because any country which refuses to agree gains huge advantage over countries which agree, and this is true for all future researched helped by super AI.

Disclaimer: I did some more digging, and I think Constitutional AI is more promising than I gave it credit for. When Claude had hidden goals, it still pursued the spirit of the constitution it was given. So deceptive alignment may not be as hard a problem as I thought, at least based empirically on recent models. I don’t know how well this will generalize to smarter models though.

I agree with your point that there could be a super AI which is also evil in the future. In fact, I think it is inevitable. My argument was simply that we can make different types of AI which are easier to control, and also, linked AI models would introduce more control. And also, we are in a situation where if we dont advance AI, someone else eventually will, same with ICBMs. I think the more we advance AI, the more we will be able to understand it, and maybe good AI will even fight against bad AI and detect bad AI if it comes to worst.

China leads in AI

China leads in cheap technology.

The U.S. leads the world in AI investment, thanks to a robust ecosystem of startups, venture capital firms and the largest publicly traded tech companies. The U.S. accounted for $109 billion in corporate AI investments in 2024 alone, which is nearly as much as the rest of the world combined.

The USA is currently the No. 1 country in AI, thanks to foundation model breakthroughs, semiconductor dominance, enterprise AI maturity, and global research leadership.

China doesnt lead anywhere except in mass production. But due to population collapse, its questionable how long will China even be world factory of cheap goods.

I did some more digging, and I think Constitutional AI is more promising than I gave it credit for. When Claude had hidden goals, it still pursued the spirit of the constitution it was given.

Claude is also very good at coding, especially when it comes to more complex coding. It outperforms chatgpt for sure, even tho chatgpt is by far fastest coder.

Concerns in theAI industry that investment in AI may not be a good return on investment. That is what drove the stock market down as well as the war in Iran.