Ash

I mean most SLM labs are doing the same thing though, go look at BananaMind, Axiomic Labs, and FromZiro. So i wasnt doing anything out of the normal, and its the same training data as Pebble-25M and Pebble-10M so it isnt 100% the dataset.

that could work, we are going to start a series of tests that will hopefully give some useful data, simplifying data will probably be part of it.

Thank you for that link, i might train future models with mamba3 right now though im going to stick with mamba2, mamba2 vs mamba3 will probably be one of the things i test in the hunt for the optimal mamba models.

I also basically live under a rock, i somehow miss a lot of things in the AI space..

tbh i didnt actually know that mamba3 was out... but thanks for letting me know.

Thank you ContextReq!

Unfortunately as far as i am am able to get the models to run, i havnt been able to get CPU working, when i do though i will let everyone know..

@CodeSoft If you want some tips for training decent models in this size range what i did/do is at least have fineweb-edu and cosmopedia-v2, and it would be recommended to have DCLM, math data like FineMath-4+, and a tiny bit of code data like python but for 5M you would probably be fine skipping code, use the Muon opimizer for 2D weights and AdamW for everything else for 5M and i would aim for at least 500M tokens, and i would settle for a 2048 vocab tokenizer just to leave the most room possible for non embedding params. I wish you the best of luck and i hope this helps

you know, i'll actually test all of this. i can run some experiments comparing:

Architecture Mamba : Attention
Mamba-only ∞:1
Mostly Mamba 7:1
Current model 3:1
More attention 1:1
Transformer-only 0:1

Then we can actually see which ratio performs best at this scale.

As far as I know, I don't think the attention layers were dominating the gradients, but I'll measure that when I run these experiments.

When I get to them, the models will probably be released under https://huggingface.co/basically-experimental

I settled on 3:1 mostly because that seems to be the standard for hybrids so i thought that would be a good starting point, future models will probably use different ratios though

@Qozimo I didn't realize using a 2048 vocab would cause this much discourse, so i guess you might not like to hear that i forgot up the context and vocab on Pebble 25M. banaxi is right though chinchilla isn't what we use in the small LM space, BananaMind2 models were trained on 33B tokens and other SLMs go above and beyond chinchilla. I guess i can up the vocab on the next generation of Pebble but if you think Pebble sucks then so be it i just want to see a 10M model from you that beats not only Pebble 10M but also BananaMind2-Nano