4 appearances on the backlist front page in the last 30 days.
Building momentum at Marin! Upgrading from Dense -> 129B parameter MoEs -> architecture improvements -> optimizer improvements gives our pretraining recipe an estimated 6x cumulative learning speedup, accounting for MFU. Includes community
In modded-nanogpt we also found that the last couple attention layers hate interacting with the final prediction MLPs. So we work around it with a cached activation from earlier. In the attention residuals paper, Kimi doesn't explicitly men
New NanoGPT Speedrun WR at 81.8 (-2.6s) from @.Lisennlp on Github with MUDD skip connections, an expressive and efficient mechanism for data dependent skips! Instead of a learned scalar or sigmoid(linear) gate, MUDD uses a 64 neuron 'MLP' t
Paper here: https:// arxiv.org/pdf/2502.12170. The MUDD coefficients are used for many purposes, such as routing multiple layers into future attention values, modulating the value embedding, modulating the bigram embedding, etc. (Delay on