[httpjson-pagination] Pagination/cursor bugs in AWS SecurityHub, AWS Inspector, Tenable SC, and Atlassian Confluence

## Findings

### 1. AWS Security Hub can fail to advance cursor when the last page has exactly 100 findings

**Location**
- `packages/aws/data_stream/securityhub_findings/agent/stream/httpjson.yml.hbs:43-47`
- `packages/aws/data_stream/securityhub_findings/agent/stream/httpjson.yml.hbs:54-56`

**Evidence**
```yaml
response.pagination:
- set:
target: body.NextToken
value: '[[if (eq (len .last_response.body.Findings) 100)]][[.last_response.body.NextToken]][[end]]'
...
cursor:
last_execution_datetime:
value: '[[if (ne (len .last_response.body.Findings) 100)]][[.last_event.UpdatedAt]][[end]]'
```

**What is wrong**
Pagination and cursor advancement are both keyed to `len(Findings) == 100` rather than actual token presence. A valid terminal response can contain exactly 100 findings and no `NextToken`.

**Runtime failure trace**
1. Poll window returns pages of 100 findings.
2. Final page is exactly 100 findings and has no `NextToken`.
3. Pagination condition still executes because length is 100, but no real continuation token exists.
4. Cursor update is skipped because `len == 100`.
5. Next interval re-queries from stale cursor, replaying already-processed findings.

**Why it matters (high severity)**
This can repeatedly re-ingest the same findings and generate sustained redundant API traffic.

**Suggested fix**
Gate pagination on `NextToken` existence/non-empty value (not page length), and advance cursor whenever a page returns findings.

---

### 2. AWS Inspector has the same exact-full-page cursor/pagination failure mode

**Location**
- `packages/aws/data_stream/inspector/agent/stream/httpjson.yml.hbs:44-48`
- `packages/aws/data_stream/inspector/agent/stream/httpjson.yml.hbs:55-57`

**Evidence**
```yaml
response.pagination:
- set:
target: body.nextToken
value: '[[if (eq (len .last_response.body.findings) 100)]][[.last_response.body.nextToken]][[end]]'
...
cursor:
last_observe_datetime:
value: '[[if (ne (len .last_response.body.findings) 100)]][[.last_event.lastObservedAt]][[end]]'
```

**What is wrong**
Identical pattern to Security Hub: continuation and cursor advancement are inferred from count instead of token semantics.

**Runtime failure trace**
1. API returns full pages of 100 findings.
2. Terminal page contains exactly 100 findings but no `nextToken`.
3. Cursor advancement is skipped because `len == 100`.
4. Next poll starts from stale `last_observe_datetime`, replaying prior window.

**Why it matters (high severity)**
Stale cursor causes repeated collection of the same records and unnecessary API load.

**Suggested fix**
Only paginate when `nextToken` is present/non-empty; update cursor whenever non-empty findings are processed.

---

### 3. Tenable SC plugin cursor is never advanced when total results are an exact multiple of `batch_size`

**Location**
- `packages/tenable_sc/data_stream/plugin/agent/stream/httpjson.yml.hbs:54-57`
- `packages/tenable_sc/data_stream/plugin/agent/stream/httpjson.yml.hbs:67-70`

**Evidence**
```yaml
response.pagination:
- set:
target: url.params.startOffset
value: '[[if (ne (len .last_response.body.response) 0)]][[toInt (.last_response.url.params.Get "endOffset")]][[end]]'
...
cursor:
last_event_ts:
value: '[[if (lt (len .last_response.body.response) \{\{batch_size}})]][[.last_event.pluginModDate]][[end]]'
ignore_empty_value: true
```

**What is wrong**
Cursor updates only when `len(response) < batch_size`. If result count is exactly `N * batch_size`, the final non-empty page does not update cursor, and the next empty page has no `.last_event` to store.

**Runtime failure trace**
1. Page 1..N each return exactly `batch_size` records.
2. Cursor update condition is false on each full page.
3. Pagination requests one more page, which is empty.
4. Empty page satisfies `< batch_size`, but `.last_event` is missing; `ignore_empty_value: true` drops the update.
5. Cursor remains stale; next interval re-reads the same range.

**Why it matters (high severity)**
This creates persistent duplicate ingestion and avoidable API hammering for large datasets.

**Suggested fix**
Set cursor from the last non-empty page (e.g., update when `len(response) > 0`) instead of only on short pages.

---

### 4. Atlassian Confluence Cloud uses a numeric cursor without string/integer reformat on replay

**Location**
- `packages/atlassian_confluence/data_stream/audit/agent/stream/httpjson.yml.hbs:38-40`
- `packages/atlassian_confluence/data_stream/audit/agent/stream/httpjson.yml.hbs:67-69`
- `packages/atlassian_confluence/_dev/deploy/docker/files/config.yml:35`

**Evidence**
```yaml
# request
- set:
target: url.params.startDate
value: '[[.cursor.last_timestamp]]'

# cursor
last_timestamp:
value: '[[add (toInt .first_event.creationDate) 1]]'
```
Mock endpoint requires numeric digits only:
```yaml
startDate: "{startDate:[0-9]+}"
```

**What is wrong**
Cursor is persisted as JSON numeric state, then replayed without explicit integer string coercion. For large millisecond values, numeric JSON round-trips can surface as floating-point representations when interpolated.

**Runtime failure trace**
1. First poll stores millisecond `creationDate` cursor (`e.g., 1643097111963`).
2. Cursor is serialized/deserialized via JSON state.
3. Next request interpolates raw `.cursor.last_timestamp` into `startDate`.
4. If represented non-integer text (e.g., scientific notation), `startDate` violates digits-only expectation.
5. Pagination/polling fails or silently stops advancing.

**Why it matters (high severity)**
A broken `startDate` parameter can stop collection after initial success, creating data gaps.

**Suggested fix**
Force integer-string rendering when reading cursor (for example `printf "%.0f"`/equivalent integer formatting) or persist cursor explicitly as a string and parse back to integer only when needed.

## Investigated and found correct

- `packages/cloudflare/data_stream/audit/agent/stream/httpjson.yml.hbs`: pagination stops on empty result set (`page` increments only while results are non-empty).
- `packages/github/data_stream/issues/agent/stream/httpjson.yml.hbs`: pagination follows `Link rel="next"` and terminates when absent.
- `packages/google_workspace/data_stream/admin/agent/stream/httpjson.yml.hbs`: uses explicit page-token presence and pagination-finished cursor state.
- `packages/atlassian_jira/data_stream/audit/agent/stream/httpjson.yml.hbs`: cursor path uses formatted timestamp strings for replay rather than raw numeric millisecond interpolation.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: httpjson and CEL Pagination and Cursor Integrity](https://github.com/elastic/integrations/actions/runs/27410140048)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jun 19, 2026, 10:42 AM UTC