aws-samples / aws-samples/sample-analytics-agent-progressive-disclosure

P0: cross-domain time axes are not aligned — marketing domain runs 220 days past the anchor, corrupting the governed layer (ROI off 2.9x), and three existing checks cannot see it

Open
#9 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
2
Avg merge
2d 10h
Merged PRs (30d)
3

Description

## Summary

The marketing domain and the fact domain were generated on **different time axes that share neither start nor end**. `channel_daily_costs` carries spend on dates up to 220 days past the snapshot anchor, `ad_campaigns` carries creation timestamps in the future, and both propagate into the **governed** layer — where they distort two of the nine frozen metrics in `knowledge/metrics/governed_metrics.md` by ~2.9x.

Three existing checks in the acceptance ladder are individually reasonable but their coverage leaves exactly this gap, so `scripts/test_all.sh` and `scripts/audit/run.py all` both report clean.

## 1. The misalignment

Anchor: `meta_snapshot.as_of_date = 2026-01-24`, `data_start = 2025-10-26`.

| Table | Range | vs anchor |
|---|---|---|
| `ad_campaigns` (dim) | 2025-11-02 → 2026-10-01 | 11 months |
| `channel_daily_costs` (fact) | 2025-11-02 → 2026-09-01 | 10 months |
| `orders` (fact) | 2025-10-26 → 2026-01-24 | **91 days** |

The marketing domain starts 7 days *after* the fact window (which is why cost data is missing for the first 7 days of orders) and ends 220 days *after* it. The likely cause is that the marketing generator used a "one year" span parameter while the fact generator used the 91-day window, and the two were never reconciled against a shared anchor.

## 2. A logical impossibility, not just a range issue

```sql
SELECT count(*) n, max(created_at) mx,
sum(CASE WHEN created_at > DATE '2026-01-24' THEN 1 ELSE 0 END) future_created
FROM ad_campaigns;
-- 50 | 2026-08-15 22:53:46 | 32
```

**32 of 50 ad campaigns (64%) have a `created_at` later than "today".** A creation timestamp cannot be in the future — this is not a definitional disagreement, it is a hard error.

## 3. Propagation into the governed layer

`channel_daily_costs` → `mart_channel_daily` (governed) → `dws_channel_weekly` (derived).

```
mart_channel_daily 2025-10-26 → 2026-09-01 514 of 1,788 rows (29%) are past the anchor
dws_channel_weekly 2025-10-27 → 2026-08-31
```

Those 514 rows carry `cost > 0` with `new_users_attributed = 0` and `gmv_attributed = 0`, because no users or orders can exist on those dates.

| Metric | As queried from the governed layer | With a date filter | Error |
|---|---|---|---|
| total cost | 1,449,872 | 504,024 | **65.2% of spend sits on impossible dates** |
| ROI | 50.51 | 145.29 | understated 2.9x |
| CAC | 13.83 | 4.81 | overstated 2.9x |

ROI and CAC are both frozen official definitions. The selling point of the governed layer is that consumers do not have to re-derive definitions — so a 2.9x error here is more costly than the same error on a raw table.

## 4. Why the acceptance ladder does not see it

Each of these checks is individually sensible; the gap is in their **coverage**:

| Check | Current scope | Gap |
|---|---|---|
| `L1_inventory.sql` L1.5 "core fact table time bounds, out of bounds if past the snapshot date" | 4 tables: `events`, `orders`, `sessions`, `users` | Every other time column in the database is unchecked — including `channel_daily_costs.date` |
| `L5_realism.sql` L5.1 mart-vs-detail daily reconciliation | `mart_daily_kpi`, `mart_daily_revenue` (both correctly bounded to 91 days) | `mart_channel_daily` and `mart_user_summary` are never reconciled against anything |
| `L4_semantics.sql` L4.5 `user_level` business rule | Asserts the literal documented rule | See the separate P1 issue — the check's guidance only warns about "similar averages", so it cannot report an inversion |

The intersection of the first two leaves `mart_channel_daily` in a blind spot: not a core fact table, and not one of the two reconciled marts.

## 5. Suggested fixes

**Data**
1. Give the generator a **single global time anchor** that every domain consumes, rather than per-domain span parameters. `scripts/gen/budget.py` already treats the window as scale-invariant (`DATA_START` / `AS_OF` / `WINDOW_DAYS`) — the marketing domain appears not to read it.
2. Regenerate the marketing domain, then rebuild `mart_channel_daily` and `dws_channel_weekly`.
3. Add a generator self-test in `scripts/gen/selftest_closures.py`: no time column in any table may exceed `AS_OF`, except columns whose semantics are validity bounds (`expire_at`, `end_date`, subscription `end_date`).

**Checkers** (worth doing even if the data is fixed, otherwise the next reload can reintroduce it)
4. Extend L1.5 to **every** time column in every table, with a semantic allowlist for validity-bound columns.
5. Extend L5.1 to reconcile **all four** mart tables.

## 6. Reproduce

```sql
-- axis comparison
SELECT 'ad_campaigns' t, min(start_date)::varchar d0, max(end_date)::varchar d1 FROM ad_campaigns
UNION ALL SELECT 'channel_daily_costs', min(date)::varchar, max(date)::varchar FROM channel_daily_costs
UNION ALL SELECT 'orders', min(placed_at)::varchar, max(placed_at)::varchar FROM orders;

-- phantom rows and metric impact
SELECT round(sum(cost),2) total_cost,
round(sum(CASE WHEN dt > DATE '2026-01-24' THEN cost ELSE 0 END),2) phantom_cost,
round(sum(gmv_attributed)/NULLIF(sum(cost),0),2) roi_as_queried,
round(sum(gmv_attributed)/NULLIF(sum(CASE WHEN dt<=DATE '2026-01-24' THEN cost ELSE 0 END),0),2) roi_correct
FROM mart_channel_daily;
```

Measured 2026-08-13 against `analytics-agent-wg` / `app_analytics` through a read-only role.

Contributor guide

Open the contributing guide

Research direction

Start with scripts/gen/budget.py and the marketing generator to trace how DATA_START, AS_OF, and WINDOW_DAYS are consumed. Read scripts/gen/selftest_closures.py, scripts/audit/run.py, L1_inventory.sql, L5_realism.sql, and scripts/test_all.sh before reproducing the reported ranges and governed metrics. Done means regenerated marketing data and rebuilt marts use one valid time axis, and the checks cover all applicable time columns and mart tables.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, sql
Domain
data-engineering, databases, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.