כתבה
arXiv cs.AI ·
Harbor Adapters ו-Harbor-Index: תשתית ומאגר נתונים מטא
Harbor Adapters and Harbor-Index: Infrastructure and a Curated Meta-Dataset for Large-Scale Agentic Evaluation
Harbor Adapters היא תשתית אחידה לבחינת ביצועים של סוכנים. היא כוללת 82 משימות קשות ומגוונות. GPT-5.5 עם Codex השיגה 28.0% הצלחה.
תקציר מקורי באנגליתarXiv:2609.04298v1 Announce Type: new Abstract: Evaluating agents on the growing number of agentic benchmarks is challenging because they often require complex environments and agent integrations. We introduce Harbor Adapters, a unified evaluation infrastructure for agentic benchmarks. Our work makes three contributions. First, we develop benchmark adapters that port more than 80 benchmarks to evaluate arbitrary agents, and validate them through rigorous code review and parity experiments. Second, we conduct a large-scale evaluation of 8 models spanning capability tiers across 54 benchmarks; every model is run with Terminus-2 and with one of 3 native harnesses. This enables a broader analysis of agent capabilities and failure modes than was previously possible. Third, we introduce Harbor-I
קרא במקור המקורי
arxiv.org
פתח כתבה מקורית