התאמה-מודעת-שינוי למודלי CLIP: לקיחת תיאום-תמונה-טקסט

תקציר מקורי באנגליתarXiv:2501.19060v4 Announce Type: replace-cross Abstract: Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes. Existing VLM-specific calibration methods mainly rely on textual features of train classes, limiting their applicability across class and distribution shifts. We propose \textbf{Shift-Aware Calibration (SAC)}, a training-free, sample-wise method that uses the discrepancy between the output logits of the original and fine-tuned CLIP as a calibration signal. We explicitly call this quantity \emph{logit shift}, since it is measured in prediction space rather than between hidden representations. SAC maps this logit shift to a positive sc