aws-samples / aws-samples/sample-kiro-user-analytics-dashboard

Dashboard silently truncates data at 1000 rows (Athena GetQueryResults not paginated)

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
40
Forks
10
PR merge metrics
No merged PRs in 30d

Description

## Summary

`fetch_data()` in `app/app.py` calls Athena's `GetQueryResults` **once** and reads only the first page. Athena returns a maximum of **1000 rows per page** and exposes more via `NextToken`. Because the token is never followed, any query returning more than ~999 data rows is **silently truncated** — no error is raised, rows are just missing.

## Reproduction

Run any query that returns >1000 rows through the app's result path. A minimal, data-independent repro using a generated sequence:

```sql
SELECT n FROM UNNEST(SEQUENCE(1,1500)) AS t(n)
```

A single `get_query_results` call returns exactly **1000 rows** (999 data + 1 header) and a **`NextToken` is present** (more data waiting, ignored). The dashboard therefore shows 999 rows instead of 1500.

## Impact (organizational scale)

Queries aggregated to a handful of rows (overall metrics, by-client, by-tier, top-10) are unaffected. The following truncate silently:

| Section | Rows returned | Truncates when |
|---|---|---|
| Credit Usage by User by Month (added in #2) | users × months | users × months > 999 (e.g. 100 users × 12 months = 1200) |
| User filter list / Credits by user / Engagement / Activity timeline | one per user | distinct users > 999 |
| Daily trends by client | dates × client types | ~11 months at 3 client types |
| Daily activity trends | one per date | ~2.7 years |

It also yields **inconsistent numbers** in one view: `Total Users` is computed server-side (`COUNT(DISTINCT userid)`) and stays correct, while the Engagement Funnel derives "All Users" from the truncated client-side list — so past 1000 users the same dashboard shows two different totals.

## Suggested fix

Follow `NextToken` in `fetch_data()` until exhausted, accumulating rows across pages. Note Athena includes the column header only on the **first** page, so the header must be stripped from page 1 only.

## Environment

- `app/app.py`, `fetch_data()`
- Current `main` (`fc4476b`)

I have a fix ready with unit tests and end-to-end verification and will open a PR referencing this issue.

Contributor guide

Open the contributing guide

Research direction

Start in app/app.py at fetch_data() and reproduce the issue with SELECT n FROM UNNEST(SEQUENCE(1,1500)) AS t(n). Review the existing unit tests and end-to-end verification mentioned in the issue; done means results include all 1500 rows, with the header handled correctly across pages.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
backend, cloud
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.