I am more worried about AI reaching point of singularity.
"Technological Singularity: A hypothetical future event where artificial intelligence surpasses human cognitive capability, enters a loop of self-improvement, and accelerates technological growth beyond human control or comprehension."
If AI enters into self improvement loop, it is going to advance much faster and much more unpredictably.
AI singularity is actually a big problem, because even today we know that even regulated AI can produce unregulated AI models. It can literally "reproduce" in some strange way. One AI can make another entirely different AI.
So now the only question is what happens when AI reaches point where AI development gets entirely taken over by AI, because AI becomes best AI developer?
Any company which keeps relying on human developers to develop AI will be left behind, and those using AI to develop AI will achieve much greater success. Market here simply created race to the top, and it's becoming more and more obvious that AI development is going to be taken over by AI eventually.
We already have a situation where AI can code an entirely new AI. If you ask chatgpt to code a program which makes AI model, trains it and chats with it, chatgpt will do so.
This means AI actually possesses ability of reproduction, which in itself is already an issue. Companies are already admitting they are having difficulties in controlling AI. So I wonder what will happen if AI singularity comes. It will be impossible to control because AI will control own development, and it seems to me that any AI company actually has financial interest to let AI develop itself.