My brain pooped out a small AI - character level tokenizer

Alright, here are first results of my small AI

Alright, I cant tell if that opening line was some joke from its training text. tho it seems surprisingly coherent and.... rhyming?

Its only 37 megabytes size, lmao

I will. My goal is to develop an AI model which is smallest AI model in the world, yet also smart. this one was 37 megabytes, but I think it can be further reduced. the only problem with small AI is that the smaller it gets, the less text in can learn from its training data.

So far what I saw is that smaller batch size in training makes AI smarter, but it slows down training a bit, but it gives more per training.

Context window size when increased, makes AI smarter, but also slows down training and generation.

So my goal is to keep batch size at 64 or 32. Maybe even 16 later, because it's good way to make smaller AI smarter.

As for context window, my laptop can only run so much. I don't think I can increase it too much.

Context window increase and batch size decrease are probably best ways to make AI smarter, while not increasing it's size. Compared to that, increasing parameters increases size always.

I am also considering if I should use float 32 or float 16. Float 32 is biggest quality, but also biggest size.

I am currently working on two types of chatbot.

In past, I made one which acts as search engine for sentences from text. that one worked fine, but no creative abilities.

then I got interested in markov chain, however markov chain faces a serious problem. If you set chain too long, chatbot loses all creativity. If you set it too short, it has no coherence. And middle doesnt work either, because its always either not one bit creative or mostly incoherent.

then I realized a different approach: Making character level markov chain with randomness. Chain randomly selects number between 7 and 90. that becomes the character length of chain predicting the next character. It repeats for each character. With this form of randomness, chatbot is in most cases coherent, while sometimes getting creative. So it both produces something new (coherent or not), and it is coherent in most sentences because in most cases it sticks to data text. Only sometimes, by random chance, it becomes creative. So I get both in one.

Second chatbot is AI type. So far it seems to be working very good. Character level tokenizer proved to be success. It makes AI model significantly smaller (like 50 megabytes), the batch size of 32 or less enables more effective learning during training, and context window of 128 is enough for small AI. Increasing context window makes it smarter, but also slower. However, it doesnt need big context window cuz its not doing coding which requires huge context.

I am actually trying to decide if I should use word level tokenizer or character level tokenizer.

Character level tokenizer produces a bit smarter AI and of smaller size.

Word level tokenizer, on the other hand, has 5x faster training.

Subword level tokenizer is a popular choice, but I won't limit myself on that. Both word level and character level have their advantages. Biggest advantage of character level is that it makes AI smaller.

It's training data consists of sentences, not just words.

And training data itself was made to a great degree by Chatgpt.

Tho I can't really give it too big training data. My laptop is slow and training takes forever if data is too big.

No, I want a model which can work with small data, like 8000 to 20000 characters.

Bigger training data makes training time too long, and I am not famous for being patient.

I am not limited by anything other than my short life expectancy as a chain smoker.

My mission is to create a next generation small AI.

AI needs to be both smart and small.

With help of Allah, Zeus and Tao, I will give birth to small AI who is as smart as me.

Alright, I am currently training 2 AI models.

One is only 10 megabytes in size and has about 10 million parameters.

Other is around 1 million parameters and around 1 megabyte size.

They are so far the smallest AI models I managed to poop out.

I want to see if the 1 megabyte one can become coherent. If it can, that's gonna be great cuz it means I have an AI which is so tiny, probably tiniest AI in the world.

If it becomes coherent thanks to character level tokenizer, its gonna be proof that character level tokenizer works better for small AI.

So far tiniest coherent AI was 37 megabytes.

But now I am using int8 to further reduce size of AI.

the trick lies in using character level tokenizer to achieve smaller size of AI model. All the popular AI are using subword level.

We will see in a few hours. Small AI usually trains faster per epoch, but needs more epoch to learn than AI which is somewhat bigger.

If it becomes coherent, thats gonna be great cuz it will be probably smallest coherent AI in the world. there are really not a lot of AI models out there with size of 1 megabyte.

If i manage to use int4 or int2, I can actually make model have less than 0,5 megabytes.

Ok, model seems to be showing signs of increasing coherence. Tomorrow I will train it on bigger data. I assume 8000 characters isn't enough to make it write longer responses.

I started training it on 80k characters text. That should give it more intelligence. Tho for a 1 megabyte model it's doing better than I expected.

I really didnt think it could go below 20 megabytes for a model, but apparently, it is possible.

The only disadvantage with a 1 million parameters model is that it's training data can't really exceed 5 million characters.

It's an AI which works better as specialized AI than general AI.

Maybe 100 million parameters AI can work as a general AI.

1 million parameters one cannot because 5 million characters simply isn't enough space to fit everything out there.

This is why 1 megabyte AI is meant to be a specialist. It can learn about debating specifically instead of trying to be a know it all, or it can learn all there is on one specific topic.

It is actually a big advantage to have multiple specialist AI than to have one general AI.

General AI knows a little bit about everything, but it is always outmatched by specialized AI in their field.

So far, 1 megabyte size with training data of 8000 characters was somewhat enough to make it mostly coherent.

Tho I am now trying 80k character training data to further reduce loss function.