antirez / antirez/ds4

ds4-server: optional --imatrix-out to collect the imatrix from live traffic?

オープン
#370 コメント 1 件 リアクション 1 件 担当者 0 名 GitHub で見る
主要言語
C
スター
22.3k
フォーク
2.1k
平均マージ
1日 3時間
マージ済み PR(30日)
4

説明

Before opening a PR I'd rather ask, since it touches the serving path.

Proposal: `ds4-server --imatrix-out FILE` reuses the existing `ds4_imatrix_collector` and the `imatrix_collect_layer_batch` hook already in `metal_graph_prefill_layer_major`, accumulating the routed-MoE imatrix from live prefills and writing it on shutdown. No new backend code — a `ds4_session` field, a couple of public wrappers, and the prefill call-sites passing the collector. Default off is byte-identical (the field stays NULL unless `--imatrix-out` is given). The file holds only per-expert second moments + counts — no prompt text, ids, or positions.

Why on the server and not just `ds4 --imatrix-dataset`: the offline path needs you to materialize a prompt dataset on disk first. When prompts can't be written to disk or leave the machine for compliance reasons — e.g. patient data under GDPR — that path isn't usable, but collecting aggregate stats in-process is, since no prompt content ever reaches disk. That's the case where the server entry point isn't redundant with the CLI.

Cost: while enabled it forces the layer-major readback (suppresses the streaming-decode fast path), so prefill is slower; decode unaffected; opt-in.

Does this fit ds4, or would you rather keep imatrix collection CLI-only?

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。