ModelEngine-Group / ModelEngine-Group/unified-cache-management

[Feature]: ToolKit

Open
#1,208 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
C++
Stars
334
Forks
119
Avg merge
1d 15h
Merged PRs (30d)
82

Description

🚀 The feature, motivation and pitch
  • UCM 部署前预检工具
    功能:在 UCM 部署前运行环境预检,验证存储带宽、内核版本、驱动版本、shm-size 等关键配置是否满足运行要求。
    输出:预检结果(PASS/FAIL)及每项检查的实际值与阈值对比,对 FAIL 项给出修复建议。
    用途:在正式部署 UCM 前快速验证环境就绪,避免因环境问题导致的部署失败或运行异常。
    落地:ucm-toolkit run precheck,由 sub-issue 跟踪。

  • KV Cache 容量计算器
    功能:输入模型目录、请求长度、请求数量及 DP/TP 数,计算产生的总 token 数与 KV Cache 总占用容量。
    输出:非 HBM 存储介质的 KV Cache 总容量,以及按 DP/TP 分摊后的每卡 KV Cache 占用量。
    用途:用于部署容量规划与压测前的资源估算,快速回答"给定模型与请求数量,KV Cache 占用多少容量"。
    落地:ucm-toolkit run kv-calc,由 sub-issue 跟踪。

  • UCM 存储带宽测试工具
    功能:输入模型目录、xPU卡数、输入序列长度、并发数、kvcache 存储目录与测试轮次,测量当前环境在 UCM 运行时保存 kvcache 到存储、从存储加载 kvcache 到主机的端到端带宽。
    输出:端到端带宽(端到端带宽=数据量/端到端耗时,端到端耗时含网络时延)。
    用途:用于评估不同输入规模场景下,UCM 使用存储链路的带宽。
    落地:ucm-toolkit run bandwidth --param_name param_value,由 sub-issue 跟踪。

  • TTFT 分析工具
    功能:输入模型目录、POSIX 带宽、H2D 带宽、输入序列长度,以及该输入长度下 Full Prefill 与 Full HBM Prefix Cache 的 TTFT 指标。
    输出:估算 Full UCM SSD/DRAM Prefix Cache 的 TTFT(单位ms),输出相对增益/损失。
    用途:用于部署前 TTFT 预判。
    落地:ucm-toolkit run ttft-analyze,由 sub-issue 跟踪。

  • 推理系统监控工具
    功能:输入模型服务 URL 与监控时间范围,刻画该时间段内 KV Cache 的运行状态。
    输出:KV Cache 的 HBM/DRAM/SSD 命中率、TTFT/TPOT/E2E/TPS 指标、输入/输出序列长度分布,以及各 KV Cache 介质之间的输出传输带宽。
    用途:运行时监控与性能分析。
    落地:ucm-toolkit run metric-view,由 sub-issue 跟踪。

Alternatives

No response

Additional context

No response

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the sub-issues referenced for each ucm-toolkit run command, especially kv-calc and ttft-analyze; the checked precheck, bandwidth, and metric-view items show work already underway. Confirm each command's inputs and outputs in its sub-issue, then use its stated validation requirements to determine when the remaining tools are complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, performance, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.