aws-samples / aws-samples/sample-analytics-agent-progressive-disclosure

L5 gate hardcodes pre-fix GMV, so test_all.sh always fails (assert the four paths agree instead)

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
2
Avg merge
2d 10h
Merged PRs (30d)
3

Description

## Problem

The L5 gate in `scripts/test_all.sh:132` hardcodes the expected GMV as `149685621.44`:

```bash
grep_run "四条独立路径 GMV 完全相等" "149685621\.44.*149685621\.44.*149685621\.44.*149685621\.44" \
python3 scripts/redshift/rsql.py "..."
```

That is the value from **before** the 2026-08-06 data fix and reload. The actual value after the reload is `151238025.32`, so **anyone running `bash scripts/test_all.sh` today gets a FAIL at L5 even though the data is correct**.

`docs/data-audit.md:490` already records the change ("GMV = **151,238,025** (was 149,685,621; +1% from end-of-window `refunded` orders being downgraded to `paid` for the remaining window)"), but the hardcoded constant in the gate was never updated.

## Measured

All four paths, queried through a read-only audit role (2026-08-11, `analytics-agent-wg` / `app_analytics`):

```
detail kpi revenue ads
------------ ------------ ------------ ------------
151238025.32 151238025.32 151238025.32 151238025.32
```

The four paths **agree exactly** — the property this check exists to verify holds. The only thing that is red is the stale constant.

## Suggestion

Please don't just swap the constant for `151238025.32`. That fixes this reload and breaks on the next one; the same trap is already on its second occurrence. **The intent of this check is "four independent paths agree with each other", not "GMV equals one specific number"**, so the assertion should compare the paths to each other and stay immune to data versions. For example, have the SQL return a verdict column:

```sql
SELECT CASE WHEN COUNT(DISTINCT v) = 1 THEN 'GMV_PATHS_AGREE' ELSE 'GMV_PATHS_DIFFER' END AS verdict,
MAX(v) AS gmv
FROM (SELECT SUM(gmv) v FROM mart_daily_kpi
UNION ALL SELECT SUM(gmv) FROM mart_daily_revenue
UNION ALL SELECT SUM(gmv) FROM growth_daily_gmv
UNION ALL SELECT SUM(actual_amount) FROM orders WHERE status IN ('paid','shipped','delivered'))
```

and let `grep_run` match `GMV_PATHS_AGREE`. The actual GMV is still printed for humans to read, it just no longer participates in the pass/fail decision.

## Related (same class of defect)

At least three hardcoded expected values were left stale by the data fix. Worth handling together, and worth adding a "what hardcoded values must be updated after a reload" checklist to `docs/test-plan-v2.md`:

1. This issue: the GMV constant at `scripts/test_all.sh:132`
2. `web/catalog.json`: generated 2026-08-04, before the 8/6 reload — says `payments` = 691,800 (actual 689,255) and 96,491,202 rows all-layers (actual 91,294,056). Filed separately.
3. `docs/data-audit.md:330`: the L5.1 output block still shows the old `149685621`. The "post-fix re-audit" table at the end of the file is correct, but this block in the body was not updated, which is easy to misread.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/test_all.sh:132 and inspect how grep_run executes the L5 SQL through scripts/redshift/rsql.py. Change the check so it reports whether the four GMV paths agree rather than matching a fixed value, then run bash scripts/test_all.sh and confirm L5 passes while still printing the GMV. Review docs/data-audit.md:330 and docs/test-plan-v2.md if handling the related stale-value documentation.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, shell, sql
Domain
data-engineering, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.