Hugging Face's logo
CodeSoft's picture
🛠️ Working on SLM Arena

CodeSoft PRO

CodeSoft
9 3 27
wayneworkman2012's profile picture arthurblg1802's profile picture MultivexAI's profile picture
·
https://codesft.dev

AI & ML interests

Working on small language models

Recent Activity

liked a model about 5 hours ago
DALabCommunity/Haidass1.5-143M
upvoted a paper about 5 hours ago
On-Policy Self-Distillation in Diffusion Models
repliedto their post about 5 hours ago
Over the past week or so, I've been working on some models, those released being https://huggingface.co/CodeSoft/sorbet-25m and https://huggingface.co/CodeSoft/sorbet-v2-25m. In general, I'm a little confused because no matter what hyperparameters I change or datasets I add/remove, the benchmarks never move up. In a recent project, where I attached a TN-gram block to Sorbet-v2-25M, it still stayed the same on benchmarks despite the TN-gram clearly learning (due to the perplexity being lower with the TN-gram attached). When I changed the corpus to favor higher density text (the first paragraphs of Wikipedia articles and synthetic math), the benchmarks either stayed flat or went down. Does anyone have ideas on what I can do to improve my models? I'd really appreciate any feedback!
View all activity

Organizations

Small Language Model Consortium's profile picture