כתבה
arXiv cs.AI ·
Mooncake: ארכיטקטורת ניתוח חילופין סנטרית ל-LLM Serving
Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving
Mooncake היא ארכיטקטורת ניתוח חילופין סנטרית ל-LLM Serving, המספקת תהודה גבוהה ומקסימום תעבורה. היא נבנתה על ידי Moonshot AI ומשמשת כפלטפורמה ל-LLM Kimi.
תקציר מקורי באנגליתarXiv:2407.00079v5 Announce Type: replace-cross Abstract: Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI. It features a KVCache-centric disaggregated architecture that separates the prefill and decoding clusters. It also leverages the underutilized CPU, DRAM, and SSD resources of the GPU cluster to implement a disaggregated cache of KVCache. The core of Mooncake is its KVCache-centric scheduler, which balances maximizing overall effective throughput while meeting latency-related Service Level Objectives (SLOs). Unlike traditional studies that assume all requests will be processed, Mooncake faces challenges due to highly overloaded scenarios. To mitigate these, we developed a prediction-based early rejection policy. Experiments show that Mooncake ex
קרא במקור המקורי
arxiv.org
פתח כתבה מקורית