aws-samples / aws-samples/sample-kiro-user-analytics-dashboard
Dashboard silently truncates data at 1000 rows (Athena GetQueryResults not paginated)
- Dominant language
- Python
- Stars
- 40
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
`fetch_data()` in `app/app.py` calls Athena's `GetQueryResults` **once** and reads only the first page. Athena returns a maximum of **1000 rows per page** and exposes more via `NextToken`. Because the token is never followed, any query returning more than ~999 data rows is **silently truncated** — no error is raised, rows are just missing.
## Reproduction
Run any query that returns >1000 rows through the app's result path. A minimal, data-independent repro using a generated sequence:
```sql
SELECT n FROM UNNEST(SEQUENCE(1,1500)) AS t(n)
```
A single `get_query_results` call returns exactly **1000 rows** (999 data + 1 header) and a **`NextToken` is present** (more data waiting, ignored). The dashboard therefore shows 999 rows instead of 1500.
## Impact (organizational scale)
Queries aggregated to a handful of rows (overall metrics, by-client, by-tier, top-10) are unaffected. The following truncate silently:
| Section | Rows returned | Truncates when |
|---|---|---|
| Credit Usage by User by Month (added in #2) | users × months | users × months > 999 (e.g. 100 users × 12 months = 1200) |
| User filter list / Credits by user / Engagement / Activity timeline | one per user | distinct users > 999 |
| Daily trends by client | dates × client types | ~11 months at 3 client types |
| Daily activity trends | one per date | ~2.7 years |
It also yields **inconsistent numbers** in one view: `Total Users` is computed server-side (`COUNT(DISTINCT userid)`) and stays correct, while the Engagement Funnel derives "All Users" from the truncated client-side list — so past 1000 users the same dashboard shows two different totals.
## Suggested fix
Follow `NextToken` in `fetch_data()` until exhausted, accumulating rows across pages. Note Athena includes the column header only on the **first** page, so the header must be stripped from page 1 only.
## Environment
- `app/app.py`, `fetch_data()`
- Current `main` (`fc4476b`)
I have a fix ready with unit tests and end-to-end verification and will open a PR referencing this issue.
Contributor guide
Research direction
Start in app/app.py at fetch_data() and reproduce the issue with SELECT n FROM UNNEST(SEQUENCE(1,1500)) AS t(n). Review the existing unit tests and end-to-end verification mentioned in the issue; done means results include all 1500 rows, with the header handled correctly across pages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- backend, cloud
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100