ADORSYS-GIS / ADORSYS-GIS/ai-helm

[Ticket]: Documentation fact-check long tail — 8 documents with known-false claims

Đang mở
#1,106 0 bình luận 0 reaction 1 người được giao Được @Koufan-De-King nhận Xem trên GitHub
ticket
Ngôn ngữ chính
Go Template
Star
3
Fork
1
Merge trung bình
19 giờ 24 phút
Pull request đã merge (30 ngày)
80

Mô tả

### Type

Documentation

### Summary

A claim-by-claim fact-check of `inference-ops` on 2026-09-03 produced ~200 findings, ~85 of them outright false. The worst were fixed the same day; **this ticket tracks what was recorded and deliberately left unfixed**, so the gap is visible rather than silent.

### Intent

The repo now states, on one page, which of its documents are known to contain false claims. That is better than pretending they are clean — but it is not a resting place. This closes the gap.

### Source of truth (links)

- Summary, method and the open list: https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/reference/doc-fact-check-2026-09-03.md
- Raw findings (~5,400 lines, `file:line` + evidence + suggested fix): `docs/reference/evidence/2026-09-03-doc-fact-check/`
- Related: ai-helm#1105 (void benchmark), ADORSYS-GIS/ai-helm-values#377 (plaintext credential)

### Current Behavior

Fixed already: the fabricated citation, `C_hbm`, ADR-0010's break-even, the `z-image-turbo` reachability error (ADR-0014 supersedes ADR-0007), the "rollout insurance" framing, three dangerous runbook procedures, 39 stale catalog paths, and ERRATA banners on three immutable reports.

Still wrong, and known:

| # | Document | Problem |
|---|---|---|
| 1 | `explanation/qwen3-5-2b-hybrid-architecture-lmcache.md` | Its central "payoff paradox" is refuted by measurement — LMCache stores both KV groups. Also misattributes a quotation to the wrong GitHub user. **Needs a rewrite, not a patch** — it is ~395 lines built on the wrong premise |
| 2 | `explanation/vram-budgeting.md` | The decode-ceiling formula reproduces none of the repo's three decode figures, one of which is now void |
| 3 | `explanation/inference-at-camer-digital.md` | "Every model reached through a single gateway" and "exactly two sources" are false; its security claim is contradicted by ai-helm-values#377 |
| 4 | `reference/qwen3-5-2b-deployment-baseline.md` | The frozen config is stale (`l1SizeGb: 8` vs live 32) and its fingerprint no longer matches the running args — so the freeze no longer freezes anything |
| 5 | `reference/observability.md` | Says the image model has no `/metrics` and there are no LMCache metrics; both exist |
| 6 | `how-to/validate-model-production-readiness.md` | P2's threshold implies ~890 tok/s against a fleet best of ~64–94; P3/P5 conflate arrival rate with concurrency |
| 7 | `benchmarks/README.md` | Indexes fewer reports than exist; one link points at a report that does not exist |
| 8 | ADRs 0002–0006 | Assorted stale external claims. **ADR-0006's premise is disproven and it is still `Proposed`, still proposing a production model swap** |

### Expected Behavior

Each row either corrected, or superseded, or explicitly retired — and the summary page updated so its "not fixed" table shrinks to empty.

### Acceptance Criteria

- [ ] Rows 1–8 each resolved, in roughly that order — 1, 4 and 8 first, because those three actively mislead a decision.
- [ ] **Row 4 first if you only do one.** A stale freeze is worse than no freeze: it invites people to believe a fingerprint that no longer matches.
- [ ] **Row 8: ADR-0006 must be resolved either way.** It proposes swapping the production coding model on a premise now known false. Either supersede it or re-argue it on the real numbers; leaving it `Proposed` is the worst option.
- [ ] The "What is NOT fixed" table in `doc-fact-check-2026-09-03.md` updated as each lands, and empty at close.
- [ ] Consider a lightweight recurring check so this does not silently rebuild.

### Out of Scope

- Re-running the fact-check itself (that is the recurring-check question, not this ticket).
- ai-helm#1105 and ai-helm-values#377 — separately tracked.
- Rewriting immutable benchmark reports; they carry ERRATA banners by design.

### Technical Context

The raw findings give `file:line`, the quoted claim, the contradicting evidence and a ready-to-paste fix, so most rows are mechanical. Rows 1 and 2 are not — they need someone to decide what the document should now say, which is why they are listed as rewrites.

The method that found these is worth reusing: ask **"what would make this conclusion false, and did anyone check that?"** rather than "is the stated evidence true?". Every large error in this repo passed the second test and failed the first.

### Risks

- Row 1 is a large rewrite of a document other docs cite; expect link churn.
- Rows 5 and 7 are cosmetic-ish and could crowd out 1/4/8. Sequence deliberately.

### Test Plan

Per row: verify against the live cluster or the primary source, fix, and re-run the repo's link check. For row 4, regenerate the fingerprint with `tools/baseline_fingerprint.sh` and confirm it matches the running container's args.

### Verification evidence

`docs/reference/evidence/2026-09-03-doc-fact-check/` — six findings files, each with a `## Summary` giving counts per verdict.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.