כתבה
arXiv cs.LG ·
Pruned BPE: ניקוי BPE: ניקוי חשיפה אחרי הלמידה והחלפת סמלים לאחר BPE
Pruned BPE: Post-training Visibility Pruning and Token Reallocation for Byte Pair Encoding
Pruned BPE מצטמצם באופן יעיל את אורך הקודקס המוקודקס באמצעות ניקוי חשיפה אחרי הלמידה והחלפת סמלים.
תקציר מקורי באנגליתarXiv:2608.00837v3 Announce Type: replace-cross Abstract: Byte Pair Encoding (BPE) is widely used for subword tokenization, but standard BPE exposes every learned merge token to the downstream model, including tokens that mainly serve as intermediate construction units and rarely appear in the final encoded corpus. This paper proposes Pruned BPE, a post-training visibility-pruning and token-reallocation method that separates merge construction from model-visible vocabulary selection. After standard BPE training, tokens are evaluated by final exposure. Low-exposure tokens are retained as internal-only merge nodes, while their visible vocabulary slots are reassigned to better-exposed candidates learned through resumed training. During encoding, internal-only tokens are recursively expanded i
קרא במקור המקורי
arxiv.org
פתח כתבה מקורית