חיי הקיימא של פעילות גדולה: ילדות סטוכסטית, גדילה תלויה בהפחתת משקל והתמקצעות תחרותית

תקציר מקורי באנגליתarXiv:2610.00423v1 Announce Type: cross Abstract: Massive activations, residual-stream coordinates with magnitudes far larger than typical activations, are associated with attention sinks in transformers, but how their scale is regulated during training remains incompletely understood. Combining training-trajectory analyses and controlled interventions, we trace their emergence, growth, and consolidation. Sink-carrying channels vary across random seeds but stabilize early within each run. Over longer training, surrounding channels erode and the sink concentrates onto a few redundant carriers. Across ablations, gradient attenuation follows the sink token's collective root-mean-square magnitude rather than any single channel, making collective scale central to understanding their effects. Ou