The main advantage of "weights based" AI models is their reasoning ability, which in some cases fails even at top models.
AI which isn't based on weights, but by pure next word prediction, can actually become creative too and write things which aren't contained in it's training data.
Because predicting next word doesn't actually equal copying whole sentences. In fact, if set to maximum randomness of next word other than following after current word somewhere in text, it will almost guaranteed produce no sentences found in its training text at all. Almost nothing it produces would match training data.
Weights based AI are extremely inefficient at a lot of what they do right now.
On one hand, they are given millions of texts to learn from.
On the other hand, they are not always learning effectively and sometimes even contradict training data.
And also, they are expensive. Their training takes long, they consume bunch of resources, and money.
Local AI are cheaper to run, but here lies the problem. Local AI are already trained. So you are getting an AI someone else trained. Unless you code it yourself, but training it is difficult because to train AI takes a lot of text.
So I came up with a very interesting solution.
An AI which is capable of not only learning from any book you give it, but also forming new sentences which aren't even found in that book.
I used to code chatbot which merely retrieves text from book.
But I realized I can actually code AI which isn't a mere copy pasting machine.
AI based on predicting next word, while keeping certain level of randomness, actually is capable of generating entirely new sentences because it produces word by word, and is able to combine parts or word groups of different sentences into one new sentence without creating a mess.
Should finish coding it tomorrow.
This is perhaps the most advanced AI I ever coded, because logic behind it is crazy. It allows both coherence and creativity, so it can make perfectly coherent new sentence not found anywhere in its training data, and at same time remain coherent thanks to context window.
I think such AI is better, because it's extremely light weight. It learns patterns entirely from groups of words, so it can combine groups into new sentences.
Because it's light weight, its much faster than any local AI, and at same time can be fed any training data in text form, any book or whatever.
While it's not as smart as AI which uses weights, it makes up for that by being more precise. It doesn't "hallucinate" in any sense of that word unless person manually increases randomness. New sentences it makes still come from word groups in it's training data. It doesnt "make something out of nothing".
I will see if I can finish this project.