כתבה
arXiv cs.LG ·
תיאור של High Bandwidth Flash ל-LLM Serving
Characterizing High Bandwidth Flash for LLM Serving
בדיקת High-Bandwidth Flash לשיפור ביצועי LLM Serving. המחקר חוקר את יעילות High-Bandwidth Flash בשיפור ביצועי LLM Serving, ומציג תוצאות חיוביות לשימוש ב-HBF לשיפור ביצועי LLM Serving.
תקציר מקורי באנגליתarXiv:2609.39131v1 Announce Type: new Abstract: Large language model (LLM) serving requires substantial memory to store model weights and KV caches. As models grow larger and contexts become longer, memory capacity and bandwidth increasingly become bottlenecks for serving performance. Agentic workloads compound this pressure through repeated interactions over growing contexts, making it increasingly important to retain KV state for reuse. High-bandwidth flash (HBF) offers a way to expand accelerator memory capacity for large language model (LLM) serving, but its access costs and limited write endurance complicate its use. We evaluate HBF for high-throughput agentic serving across system design and scheduling choices to understand when additional capacity improves serving performance and en
קרא במקור המקורי
arxiv.org
פתח כתבה מקורית