כתבה
arXiv cs.LG ·
SG-Blend: אינטרפולציה בין Swish ו-GELU
SG-Blend: Learning an Interpolation Between Improved Swish and GELU for Robust Neural Representations
SG-Blend הוא אלגוריתם חדש שמשלב בין Swish ו-GELU, המאפשר התאמה של פעילות הרשת לשכבות שונות. הוא הוכח כיעיל במודלים כמו BERT ו-WikiText103.
תקציר מקורי באנגליתarXiv:2505.23942v2 Announce Type: replace Abstract: Prevailing activation functions such as Swish and GELU tend toward domain-specific optima, Swish was discovered via neural architecture search on vision benchmarks, while GELU dominates transformer-based language models, and neither offers any mechanism to adapt its gating shape to individual layers. This rigidity is especially consequential in transformer FFN blocks, where LayerNorm, unlike BatchNorm, does not suppress the gradient pathologies that activation choice induces across depth. We propose SG-Blend, a per layer adaptive activation that combines SSwish, a bias-corrected, parametric Swish variant we also introduce, with learnable sharpness \b{eta} and zero-centering bias {\gamma}, with GELU through a per-layer blend coefficient {\
קרא במקור המקורי
arxiv.org
פתח כתבה מקורית