ADORSYS-GIS / ADORSYS-GIS/ai-helm
[Ticket]: Documentation fact-check long tail — 8 documents with known-false claims
- Ngôn ngữ chính
- Go Template
- Star
- 3
- Fork
- 1
- Merge trung bình
- 19 giờ 24 phút
- Pull request đã merge (30 ngày)
- 80
Mô tả
### Type
Documentation
### Summary
A claim-by-claim fact-check of `inference-ops` on 2026-09-03 produced ~200 findings, ~85 of them outright false. The worst were fixed the same day; **this ticket tracks what was recorded and deliberately left unfixed**, so the gap is visible rather than silent.
### Intent
The repo now states, on one page, which of its documents are known to contain false claims. That is better than pretending they are clean — but it is not a resting place. This closes the gap.
### Source of truth (links)
- Summary, method and the open list: https://github.com/ADORSYS-GIS/inference-ops/blob/main/docs/reference/doc-fact-check-2026-09-03.md
- Raw findings (~5,400 lines, `file:line` + evidence + suggested fix): `docs/reference/evidence/2026-09-03-doc-fact-check/`
- Related: ai-helm#1105 (void benchmark), ADORSYS-GIS/ai-helm-values#377 (plaintext credential)
### Current Behavior
Fixed already: the fabricated citation, `C_hbm`, ADR-0010's break-even, the `z-image-turbo` reachability error (ADR-0014 supersedes ADR-0007), the "rollout insurance" framing, three dangerous runbook procedures, 39 stale catalog paths, and ERRATA banners on three immutable reports.
Still wrong, and known:
| # | Document | Problem |
|---|---|---|
| 1 | `explanation/qwen3-5-2b-hybrid-architecture-lmcache.md` | Its central "payoff paradox" is refuted by measurement — LMCache stores both KV groups. Also misattributes a quotation to the wrong GitHub user. **Needs a rewrite, not a patch** — it is ~395 lines built on the wrong premise |
| 2 | `explanation/vram-budgeting.md` | The decode-ceiling formula reproduces none of the repo's three decode figures, one of which is now void |
| 3 | `explanation/inference-at-camer-digital.md` | "Every model reached through a single gateway" and "exactly two sources" are false; its security claim is contradicted by ai-helm-values#377 |
| 4 | `reference/qwen3-5-2b-deployment-baseline.md` | The frozen config is stale (`l1SizeGb: 8` vs live 32) and its fingerprint no longer matches the running args — so the freeze no longer freezes anything |
| 5 | `reference/observability.md` | Says the image model has no `/metrics` and there are no LMCache metrics; both exist |
| 6 | `how-to/validate-model-production-readiness.md` | P2's threshold implies ~890 tok/s against a fleet best of ~64–94; P3/P5 conflate arrival rate with concurrency |
| 7 | `benchmarks/README.md` | Indexes fewer reports than exist; one link points at a report that does not exist |
| 8 | ADRs 0002–0006 | Assorted stale external claims. **ADR-0006's premise is disproven and it is still `Proposed`, still proposing a production model swap** |
### Expected Behavior
Each row either corrected, or superseded, or explicitly retired — and the summary page updated so its "not fixed" table shrinks to empty.
### Acceptance Criteria
- [ ] Rows 1–8 each resolved, in roughly that order — 1, 4 and 8 first, because those three actively mislead a decision.
- [ ] **Row 4 first if you only do one.** A stale freeze is worse than no freeze: it invites people to believe a fingerprint that no longer matches.
- [ ] **Row 8: ADR-0006 must be resolved either way.** It proposes swapping the production coding model on a premise now known false. Either supersede it or re-argue it on the real numbers; leaving it `Proposed` is the worst option.
- [ ] The "What is NOT fixed" table in `doc-fact-check-2026-09-03.md` updated as each lands, and empty at close.
- [ ] Consider a lightweight recurring check so this does not silently rebuild.
### Out of Scope
- Re-running the fact-check itself (that is the recurring-check question, not this ticket).
- ai-helm#1105 and ai-helm-values#377 — separately tracked.
- Rewriting immutable benchmark reports; they carry ERRATA banners by design.
### Technical Context
The raw findings give `file:line`, the quoted claim, the contradicting evidence and a ready-to-paste fix, so most rows are mechanical. Rows 1 and 2 are not — they need someone to decide what the document should now say, which is why they are listed as rewrites.
The method that found these is worth reusing: ask **"what would make this conclusion false, and did anyone check that?"** rather than "is the stated evidence true?". Every large error in this repo passed the second test and failed the first.
### Risks
- Row 1 is a large rewrite of a document other docs cite; expect link churn.
- Rows 5 and 7 are cosmetic-ish and could crowd out 1/4/8. Sequence deliberately.
### Test Plan
Per row: verify against the live cluster or the primary source, fix, and re-run the repo's link check. For row 4, regenerate the fingerprint with `tools/baseline_fingerprint.sh` and confirm it matches the running container's args.
### Verification evidence
`docs/reference/evidence/2026-09-03-doc-fact-check/` — six findings files, each with a `## Summary` giving counts per verdict.
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.