aws-samples / aws-samples/sample-analytics-agent-progressive-disclosure
Three verification assets are not wired into the acceptance ladder: audit/run.py absent from L0-L8, cases_traps.json is dead code, manifest --check uncalled
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 3
Description
## Problem
Three verification assets are not reachable from any script or documented path, which breaks the assumption that "finishing the test plan means the work has been verified."
### 1. `scripts/audit/run.py` is absent from the L0–L8 ladder
This is the deepest data-quality tool in the repo (32 statements, L1 inventory → L5 distribution realism) and it is the instrument that originally found the 3 P0 and 5 P1 defects. Yet it appears **neither in `scripts/test_all.sh` nor in any layer of `docs/test-plan-v2.md`** — it is documented only in the "how to run" section of `docs/data-audit.md`.
Consequence: a reviewer who works through L0–L8 in `docs/test-plan-v2.md` will believe verification is complete, having never run the one layer that can surface problems like "funnel converts at ~100%" or "retention does not decay." L0 only covers the generator self-tests (40 cross-table assertions at scale=1, in memory) — it does not verify **the 91.29M rows actually in the warehouse**.
Suggestion: add a layer to the test plan (e.g. L3.5, or L9 "manual data-quality review") stating explicitly that it is **human-read, has no PASS/FAIL**, and does not belong in the automated gate — plus why (L5 requires comparing distribution shapes against real-world business experience, which is not machine-decidable). At minimum, make it visible in the ladder.
### 2. `eval/cases_traps.json` is dead code
The three level-6 trap cases (`TRAP-total-orders`, `TRAP-fin-net-revenue-dec`, `TRAP-roi-tmp-table`) are **referenced by no script** — `eval/run_eval.py:39` hardcodes `CASES_PATH = HERE / "cases.json"` with no override flag.
This is a shame because the trap tables (`orders_backup_20251201`, `tmp_campaign_roi_analysis`) are the entire point of the `noise` layer in `schema_manifest.yaml`, and the `eval_trap` field is filled in on 5 manifest entries — yet **the cases that verify whether the agent actually avoids the traps have never executed**.
Suggestion: add a `--cases ` flag to `run_eval.py` (or merge the trap cases into `cases.json` tagged as level 6) and include them in L7.
### 3. `scripts/manifest/render.py --check` and `scripts/genlib/selftest.py` belong to no layer
`render.py --check` describes itself as "for CI" in its own docstring (validating required manifest fields, the `layer` enum, `owner` required for `ads`, `replaced_by` required when `deprecated`, and `domain` pointing at an existing directory), but nothing calls it. It runs in seconds with no cloud dependency, so it belongs in L0.
If `scripts/genlib/selftest.py` (17 self-tests for the v1 generator library) is legacy along with v1, consider adding a LEGACY banner at the top of the file, consistent with what `database/00_schema_overview.md` does — otherwise reviewers can't tell whether to run it or ignore it.
## Also: minor doc/code drift
- `docs/test-plan-v2.md:49` says `selftest_fillers.py` has "26 assertions"; the source has 28 `check()` call sites, 27 of which execute (`:152-158` is a mutually exclusive try/except pair)
- `docs/test-plan-v2.md:55` says "expect **both** commands to print `全部通过`" but `:49-51` lists three commands
- `scripts/ui/render_test_prod.mjs` is executed by `scripts/test_all.sh:189-191` but the L6 section of `docs/test-plan-v2.md` documents only `render_test.mjs`, so the plan understates what that gate actually covers (the production 403-fallback path)
## Environment
All four L0 checks were run locally (Python 3.14 + numpy/pyyaml) and pass: `selftest_fillers.py` → 全部通过; `selftest_closures.py` → 全部通过(40 项断言); `pg_to_redshift.py --selftest` → 全部通过; `render.py --check` → manifest OK:8 张派生表.
Contributor guide
Research direction
Start by reading scripts/test_all.sh, docs/test-plan-v2.md, eval/run_eval.py, and the manifest validation code in scripts/manifest/render.py. Trace how the existing checks and evaluation cases are invoked, then update the documented or automated layers so the named assets are reachable and the documented assertion counts and commands match the sources. Done means the relevant checks run from their intended layer and the test plan accurately describes them.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100