[httpjson-pagination] AWS GuardDuty/Inspector cursors fail to advance when terminal page equals page size

## Findings

### 1. Cursor advancement is incorrectly tied to page length, causing replay windows when the last page is full

**Severity:** High
**Affected templates:**
- `packages/aws/data_stream/guardduty/agent/stream/httpjson.yml.hbs`
- `packages/aws/data_stream/inspector/agent/stream/httpjson.yml.hbs`

#### Evidence

**GuardDuty**
- `packages/aws/data_stream/guardduty/agent/stream/httpjson.yml.hbs:40-44`
```yaml
response.pagination:
- set:
target: body.nextToken
value: '[[if (ne .last_response.body.nextToken "")]][[.last_response.body.nextToken]][[end]]'
```
- `packages/aws/data_stream/guardduty/agent/stream/httpjson.yml.hbs:83-85`
```yaml
cursor:
last_execution_datetime:
value: '[[$f := (index .last_response.body "findings")]][[if $f]][[if (ne (len $f) 50)]][[.last_event.updatedAt]][[end]][[end]]'
```

**Inspector**
- `packages/aws/data_stream/inspector/agent/stream/httpjson.yml.hbs:39-43`
```yaml
response.pagination:
- set:
target: body.nextToken
value: '[[if (eq (len .last_response.body.findings) 100)]][[.last_response.body.nextToken]][[end]]'
```
- `packages/aws/data_stream/inspector/agent/stream/httpjson.yml.hbs:50-53`
```yaml
cursor:
last_observe_datetime:
value: '[[if (ne (len .last_response.body.findings) 100)]][[.last_event.lastObservedAt]][[end]]'
```

#### Runtime failure trace

1. Poll N requests a time window (`updatedAt`/`lastObservedAt` lower bound from cursor, upper bound at current time).
2. API returns the terminal page with exactly page size items (50 for GuardDuty, 100 for Inspector) and no next token (legitimate when total % page_size == 0).
3. Pagination ends (no usable `nextToken`), but cursor update condition does **not** run because it is gated on `len != page_size`.
4. Cursor remains stale.
5. Poll N+1 reuses the old lower bound and a new upper bound, replaying already-collected data and producing duplicates/redundant API load.

#### Why this is wrong

Cursor progression should depend on pagination completion / final processed record, not on `len(last_page) != page_size`. A full terminal page is valid and must still advance the cursor.

#### Why it matters

This violates exactly-once collection guarantees and can continuously replay historical windows (duplicate ingestion + sustained unnecessary API pressure).

#### Suggested fix

For both templates, decouple cursor advancement from page length:

- Advance cursor when pagination has finished (no next token available) using the last processed event timestamp.
- Alternatively, track explicit pagination completion state and set cursor once the final page is processed, regardless of page length.

## Configurations investigated and found correct

1. `packages/github/data_stream/audit/agent/stream/httpjson.yml.hbs` — cursor value is normalized with `toInt` before `parseTimestampMilli` and pagination follows RFC5988 `rel=next`; no termination/cursor-stall issue found.
2. `packages/ti_rapid7_threat_command/data_stream/vulnerability/agent/stream/httpjson.yml.hbs` — pagination depends on `nextOffset` presence and carries the same bounded time window parameters through pages.
3. `packages/ti_eset/data_stream/url/agent/stream/httpjson.yml.hbs` — pagination is explicitly gated by `body.more`, so terminal pages stop correctly.

## Suggested actions

- [ ] Update GuardDuty cursor logic so full terminal pages still advance cursor.
- [ ] Update Inspector cursor logic so full terminal pages still advance cursor.
- [ ] Add regression tests covering terminal page with `len == page_size` and `nextToken` absent, asserting cursor advances and next poll does not replay prior window.

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: httpjson and CEL Pagination and Cursor Integrity](https://github.com/elastic/integrations/actions/runs/24882635451)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on May 1, 2026, 9:46 AM UTC