89.
New research from our lab: do the "experts" inside a frontier Mixture-of-Experts model form real, separable modul…
New research from our lab: do the "experts" inside a frontier Mixture-of-Experts model form real, separable modules, a math expert, an Arabic expert, a code expert? The tempting assumption: If experts specialize into clean modules you coul