38.
The community still hasn’t fully absorbed that you can train a 30B MoE on 10T tokens, from scratch, for just 500K
The community still hasn’t fully absorbed that you can train a 30B MoE on 10T tokens, from scratch, for just 500K The big caveat is you need to get data, infra etc right from day 1. Once this ecosystem of enablers is working in unison, own