aws-samples / aws-samples/sample-analytics-agent-progressive-disclosure
L5 gate hardcodes pre-fix GMV, so test_all.sh always fails (assert the four paths agree instead)
- Dominant language
- Python
- Stars
- 1
- Forks
- 2
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 3
Description
## Problem
The L5 gate in `scripts/test_all.sh:132` hardcodes the expected GMV as `149685621.44`:
```bash
grep_run "四条独立路径 GMV 完全相等" "149685621\.44.*149685621\.44.*149685621\.44.*149685621\.44" \
python3 scripts/redshift/rsql.py "..."
```
That is the value from **before** the 2026-08-06 data fix and reload. The actual value after the reload is `151238025.32`, so **anyone running `bash scripts/test_all.sh` today gets a FAIL at L5 even though the data is correct**.
`docs/data-audit.md:490` already records the change ("GMV = **151,238,025** (was 149,685,621; +1% from end-of-window `refunded` orders being downgraded to `paid` for the remaining window)"), but the hardcoded constant in the gate was never updated.
## Measured
All four paths, queried through a read-only audit role (2026-08-11, `analytics-agent-wg` / `app_analytics`):
```
detail kpi revenue ads
------------ ------------ ------------ ------------
151238025.32 151238025.32 151238025.32 151238025.32
```
The four paths **agree exactly** — the property this check exists to verify holds. The only thing that is red is the stale constant.
## Suggestion
Please don't just swap the constant for `151238025.32`. That fixes this reload and breaks on the next one; the same trap is already on its second occurrence. **The intent of this check is "four independent paths agree with each other", not "GMV equals one specific number"**, so the assertion should compare the paths to each other and stay immune to data versions. For example, have the SQL return a verdict column:
```sql
SELECT CASE WHEN COUNT(DISTINCT v) = 1 THEN 'GMV_PATHS_AGREE' ELSE 'GMV_PATHS_DIFFER' END AS verdict,
MAX(v) AS gmv
FROM (SELECT SUM(gmv) v FROM mart_daily_kpi
UNION ALL SELECT SUM(gmv) FROM mart_daily_revenue
UNION ALL SELECT SUM(gmv) FROM growth_daily_gmv
UNION ALL SELECT SUM(actual_amount) FROM orders WHERE status IN ('paid','shipped','delivered'))
```
and let `grep_run` match `GMV_PATHS_AGREE`. The actual GMV is still printed for humans to read, it just no longer participates in the pass/fail decision.
## Related (same class of defect)
At least three hardcoded expected values were left stale by the data fix. Worth handling together, and worth adding a "what hardcoded values must be updated after a reload" checklist to `docs/test-plan-v2.md`:
1. This issue: the GMV constant at `scripts/test_all.sh:132`
2. `web/catalog.json`: generated 2026-08-04, before the 8/6 reload — says `payments` = 691,800 (actual 689,255) and 96,491,202 rows all-layers (actual 91,294,056). Filed separately.
3. `docs/data-audit.md:330`: the L5.1 output block still shows the old `149685621`. The "post-fix re-audit" table at the end of the file is correct, but this block in the body was not updated, which is easy to misread.
Contributor guide
Research direction
Start with scripts/test_all.sh:132 and inspect how grep_run executes the L5 SQL through scripts/redshift/rsql.py. Change the check so it reports whether the four GMV paths agree rather than matching a fixed value, then run bash scripts/test_all.sh and confirm L5 passes while still printing the GMV. Review docs/data-audit.md:330 and docs/test-plan-v2.md if handling the related stale-value documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, shell, sql
- Domain
- data-engineering, testing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100