4 appearances on the backlist front page in the last 30 days.
Some of the more puzzling unpublished observations from our paper: deep attention layers hate the residual stream of V and love it for QK, but if it has to make a choice, it will satisfy V over QK. Translated to finding: if we learn coeffi
In our paper, we also find another interesting angle to see how much deep attention layers hate to compute from what is in their residual stream: If you learn coefficients for standard value vectors in final attention layers, they will be
Thanks Lucas! Yes we prefetch the full V in deep layers. The early layers need standard V computation, so when they are doing computation we already know the complete list of token indices to prefetch from.
In our new paper, we naturally derive a new attention variant based on the surprising finding that deep layers benefit the most from learning a context-free value vectors, without the input from the residual stream. The attention variant: