כתבה
arXiv cs.LG ·
אופטימיזציה של מודלי שפה גדולים עם LMOs רצפיים
Optimizing Large Language Models with Chained LMOs
אופטימיזציה של מודלי שפה גדולים עם LMOs רצפיים. חידוש: TensorChain, חברת Qwen3, ומודלי שפה גדולים.
תקציר מקורי באנגליתarXiv:2610.10975v1 Announce Type: new Abstract: Muon has motivated a growing family of optimizers that compose multiple matrix normalizations, but these methods remain fragmented and lack a unified perspective. We introduce chained linear minimization oracles (chained LMOs), which cast these methods as compositions of LMOs. Despite their empirical success, many chains fall outside the standard LMO framework and can diverge on smooth convex objectives. To explain why composition can nevertheless help, we turn to linear associative memory and show that chaining can improve over Muon under anisotropic embeddings. Empirically, we propose TensorChain, a novel optimizer within the framework that stacks compatible weight matrices across different layers and normalizes the 3d tensor across its axe
קרא במקור המקורי
arxiv.org
פתח כתבה מקורית