45.
The teacher distribution matters a ton - just because we expect it to be "better" or "more knowledgeable" doesn't…
The teacher distribution matters a ton - just because we expect it to be "better" or "more knowledgeable" doesn't mean it's good to distill its distribution. We show the blindspots of vanilla context distillation, and how it can hurt rather