antirez / antirez/ds4

ds4-server: optional --imatrix-out to collect the imatrix from live traffic?

Đang mở
#370 1 bình luận 1 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
C
Star
22.3k
Fork
2.1k
Merge trung bình
1 ngày 3 giờ
Pull request đã merge (30 ngày)
4

Mô tả

Before opening a PR I'd rather ask, since it touches the serving path.

Proposal: `ds4-server --imatrix-out FILE` reuses the existing `ds4_imatrix_collector` and the `imatrix_collect_layer_batch` hook already in `metal_graph_prefill_layer_major`, accumulating the routed-MoE imatrix from live prefills and writing it on shutdown. No new backend code — a `ds4_session` field, a couple of public wrappers, and the prefill call-sites passing the collector. Default off is byte-identical (the field stays NULL unless `--imatrix-out` is given). The file holds only per-expert second moments + counts — no prompt text, ids, or positions.

Why on the server and not just `ds4 --imatrix-dataset`: the offline path needs you to materialize a prompt dataset on disk first. When prompts can't be written to disk or leave the machine for compliance reasons — e.g. patient data under GDPR — that path isn't usable, but collecting aggregate stats in-process is, since no prompt content ever reaches disk. That's the case where the server entry point isn't redundant with the CLI.

Cost: while enabled it forces the layer-major readback (suppresses the streaming-decode fast path), so prefill is slower; decode unaffected; opt-in.

Does this fit ds4, or would you rather keep imatrix collection CLI-only?

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.