Post
74
We trained a 10.9M byte-level recurrent Transformer on L3 and L6. (Loop 3 and Loop 6)
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
appvoid/bet-10m
Yet L4/L5 improved too, L8 held up, and the L3→L6 gain grew during training.
Same weights. More compute. Better predictions.
This is a new architecture for effective compute after several steps beyond original training!
We mixed and matched components like time and mhc into an ouro-like byte-level language model and the result is BET, a byte-level step-elastic transformer that can run computation steps without significant degradation.
One of the coolest parts of this training was discovering how Gradient Descent decided to use the first layer as what we would consider a scratchpad! Totally destroyed for the decoder but somehow makes total sense for the next layer!
I believe looped-transformers are the future of edge computing and this is a first step towards it.
Blogpost: https://medium.com/@appvoidofficial/byte-level-elasticity-182fe2ed1d2f
appvoid/bet-10m