כתבה
arXiv cs.LG ·
פרקטיקה תאורטית להכשרה מסוג Masked Pretraining (MPT)
A Theoretical Framework for Masked Pretraining (MPT)
נוסחאות חדשות להכשרה מסוג Masked Pretraining (MPT) להבנת תפקודה. המאמר עוסק בהכשרה עצמית של MPT ובשיפורים במשימות רב-תפקודיות.
תקציר מקורי באנגליתarXiv:2609.06460v1 Announce Type: new Abstract: Recently, Masked Pretraining (MPT) based on reconstruction pretraining tasks has risen to a promising self-supervised learning paradigm across various domains and achieves remarkable performance in multiple downstream tasks. However, the theoretical understanding of the working mechanism behind MPT is still limited. In this paper, we introduce a new theoretical framework to analyze MPT and understand the crucial role of masking in extracting meaningful representations. We establish theoretical connections between MPT and another popular self-supervised paradigm: contrastive learning. We prove that the masking technique implicitly creates positive pairs that are semantically similar and the reconstruction loss pulls them together in the featur
קרא במקור המקורי
arxiv.org
פתח כתבה מקורית