My brain pooped out a small AI - character level tokenizer

these are first results of 1 megabyte AI trained on 8000 characters data.

It seems to be coherent.

I will work on making user interface a bit nicer, which is easy. I don't usually work on making user interface first because it's a waste of time if AI ends up not working properly, which happens often. I use AI for coding, and AI does sometimes give me crappy code or inserts things I didn't ask for.

Next I am doing word level tokenizer 1 million parameters.

Word level is much faster than character level, but it is also a bit dumber.

However, it enables being trained on millions of characters.

Alright, I did some calculations, and it seems word level and subword level tokenizers cant make AI of 1 million parameters.

cuz equation is "128 x vocabulary size" at bare minimum.

English language has 1 million words and at minimum 20000 subwords. If you use very small training data, you could reduce it to 5000 words, but its still too much.

Any large training data will produce vocabulary of at least 5,000 to 10,000 subwords, thus 640,000 to 1,280,000 parameters for vocabulary alone.

So the only one capable of having 1 million parameters and trained on large data is character level tokenizer I made so far.

Subword level can maybe be made from 3 million parameters AI model.

However, smallest possible AI model remains character level tokenizer type.

You can see AI content in post 31. Further content is gonna have to wait a bit. I have some nice training data now, and AI needs to learn it. Especially about saying "hello" and such casual responses which weren't in it's previous training data. I am also gonna try teach it math. I wonder how far can a 1 megabyte AI be pushed before it reaches its limit. I just wonder how smart it actually is. Is it just parroting training data or is it a bit more creative than that? That's the question which can best be answered by making it do math after being trained on it.

If I succeed in making world's tiniest but fully coherent AI, its gonna prove character level tokenizer is key to AI superiority. It will open way to superior AI intelligence.

All the current popular AI use subword level tokenizer. But that essentially harms their AI. By making vocabulary size bigger, they are leaving less room for AI intelligence.

Character level AI is always gonna have more trainable parameters than subword level AI of same size. thats just math. Bigger vocabulary takes away more parameters.

And when it comes to very small AI models, that difference becomes huge. Having AI of 1 million parameters with subword tokenizer can mean AI only has 200,000 trainable parameters if training data size is big.

Even chatgpt has vocabulary size of over 100,000 tokens. that means such vocabulary gotta take away at bare minimum 20 million parameters, but usually at least 200 million up to 1 billion with bigger models which have more embedding size.

Character level tokenizer reduces vocabulary size to mere 70 to 100 in english language. It reduces vocabulary by at least 1000 times. But it does not take away any AI abilities. Characters in english language can still be used to make any word or sentence. Compared to subword tokenizer, character level tokenizer is much superior in size.

Its like teaching alphabet in school. No one in school learns subword level vocabulary. It would be terribly unpractical and too difficult to use for humans. I assume AI also has it easier using character level than subword level.

My AI runs on my laptop.

But yeah, supercomputers use a lot of electricity, and AI meant to respond to 200 million prompts a day needs more than one supercomputer.

You are contributing to the energy drain.

Nah, My laptop doesn't use much electricity, any more than any other laptop. I don't have a supercomputer. If I did, then maybe my electricity bill would be higher.

What do you hope to achieve with AI?

I need AI which is smart and creative. I don't need AI which merely repeats training data. I need AI which uses training data to make something new which is coherent. So I need AI which can innovate.

Well, innovation is a simple wish, but takes a lot of effort to realize.

Not as important as having a Chatbot which can process large amount of text for me in a smart way. It makes books much easier to read.

Right now I also made Non-AI Chatbot model called infini 1.

What's special about this is that it doesn't actually repeat text. It stores patterns, and is capable of combining two sentences into one while being coherent. Also, it is perfectly capable of deductive reasoning.

You could tell it: "A = B, B = C" and it will conclude "A = C" even if that isn't in it's knowledge, thanks to forward going deduction chain which I programmed in it.

It's not AI, but it is capable.

As for AI models, I have a new one there too. This guy is only 0,6 million parameters, standing at only 1.4 megabytes size (or around 2 megabytes in float32).

What's special about this is that I also added optimizer save state function. So it has improved training.

For AI, once it's done training, maybe a day or two.

For non-AI Chatbot infini models, sure I can show it tomorrow. Gotta sleep now.

Ok, here is a quick peek at Infini 3 chatbot model, the one not using AI, but rather something else entirely.

What is special about Infini 3 chatbot is that it combines sentences based on shared patterns.

For example:

Sentences:

Cat likes milk.

Milk is healthy.

Chatbot detects a repeated pattern "milk". So sentence one is X milk. Sentence two is Milk Y.

It combines them by X Milk, Milk Y, so output becomes:

Cat likes milk, milk is healthy.

this simple "repetition, change" pattern can also be used to detect sentence structures, and make new sentences.

For example, if sentence 1 is: Cat likes milk, sentence 2 is: Cat likes playing,

then chatbot detects a pattern: "Cat likes X".

then it can replace X with something else.

Also, not to disappoint you, I promised a chatbot which can do deductive reasoning. this is my most perfect model in that area, Deducer 1.

Unlike my other codes, this code doesnt run on any "knowledge" file. It works entirely on user input.

If you say "cat = animal", "and "animal = eats food",

it will deduce "cat = eats food".

However, it only uses forward chain deduction.

So if you say "cat = animal", "dog = animal", it will not deduce "cat = dog".

Deducer can do basic math as long as math is written properly. It can do math equations which have "=" in them. It doesnt work without "=" because thats how it knows when to do forward chain deduction.