aws-samples / aws-samples/sample-analytics-agent-progressive-disclosure

Proposal: replace the Redshift Serverless data layer with Athena + S3 Tables (Iceberg)

Open
#13 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1
Forks
2
Avg merge
2d 10h
Merged PRs (30d)
3

Description

## Context

Per CONTRIBUTING.md, opening an issue first to discuss significant work before sending PRs.

The current data layer (v2, commit 8c3bc7d) is Redshift Serverless + Glue Data Catalog. I have a working implementation
that replaces it with Athena + S3 Tables (Iceberg), plus a related fix to the knowledge-card verification layer. I would
like to contribute both back.

## What the change does

**Data layer.** 48 lake tables on S3 Tables / Iceberg. `load.py` loads 35 of them; the mart / derived layer is produced by
`INSERT INTO ... SELECT` in `database/iceberg/02_mart.sql`. Queries go through Athena, so the generators needed a Trino
dialect pass (`::` casts, `DISTINCT ON`, date arithmetic like `date + 7`). The read-only SQL boundary in `backend/db.py`
is preserved unchanged.

**Verification layer.** While migrating I found a defect class rather than a single bug: the knowledge-card checker
silently skipped three ways a literal can be declared — enums inside fenced code blocks, enums on array-typed columns, and
literals written inline in a column-description field. Because the skip was silent, a card could declare `'China'` while
the lake actually holds a constant `'中国'`, and nothing would ever report it.

The fix adds a degenerate-column check that uses `min`/`max` instead of `count(DISTINCT ...)` — equivalent for deciding
"is this column constant", much cheaper on Athena, and it returns the offending constant value itself. Exemption ledgers
now carry explicit clearing conditions, so an entry that later gets fixed turns red instead of staying permanently
suppressed.

## Proposed PR split

Rather than one large PR, I would like to split it so each is reviewable:

1. **Data layer migration** — `database/`, `infra/`, `backend/db.py`, `scripts/lakehouse/`, `scripts/gen/`,
`analyticsagent/app/analytics/`.
2. **Knowledge-card accuracy + verification layer** — card corrections, the new degenerate-column check, and the added
assertions. Based on (1).

Docs would follow in a third PR, updating both `README.md` and `README.zh-CN.md` to stay consistent with the existing
bilingual layout.

## Testing

Local self-test suite is currently 53 passing / 0 failing, with 102 assertions in `selftest_closures.py` and no
counterexamples.

## Question

Does this proposed split work for you, or would you prefer the scope adjusted before I proceed?

Contributor guide

Open the contributing guide

Research direction

Start with CONTRIBUTING.md, then review the proposed areas in database/, infra/, backend/db.py, scripts/lakehouse/, scripts/gen/, and analyticsagent/app/analytics/. Read database/iceberg/02_mart.sql and selftest_closures.py first to understand the migration and verification checks. Done means the maintainers agree on the PR split and scope before implementation proceeds.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python, sql
Domain
backend, cloud, data-engineering, databases
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.