55.
New paper on activation mixing. The authors evaluate several mixing strategies across both classical FFNs and Swi… (t.co)
New paper on activation mixing. The authors evaluate several mixing strategies across both classical FFNs and SwiGLU FFNs, with ablations on dense and MoE models. One interesting result: the most expressive mixing strategy isn’t the best ch