SG-Blend: אינטרפולציה בין Swish ו-GELU

תקציר מקורי באנגליתarXiv:2505.23942v2 Announce Type: replace-cross Abstract: Prevailing activation functions such as Swish and GELU tend toward domain-specific optima, Swish was discovered via neural architecture search on vision benchmarks, while GELU dominates transformer-based language models, and neither offers any mechanism to adapt its gating shape to individual layers. This rigidity is especially consequential in transformer FFN blocks, where LayerNorm, unlike BatchNorm, does not suppress the gradient pathologies that activation choice induces across depth. We propose SG-Blend, a per layer adaptive activation that combines SSwish, a bias-corrected, parametric Swish variant we also introduce, with learnable sharpness \b{eta} and zero-centering bias {\gamma}, with GELU through a per-layer blend coeffici