elastic / elastic/integrations
[httpjson-pagination] Checkpoint Harmony Endpoint CEL cursor does not advance after empty query completion
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
## Findings
### 1. Check Point Harmony Endpoint streams re-query the same start boundary forever after an empty (`Done`) result
**Severity:** High
**Affected templates:**
- `packages/checkpoint_harmony_endpoint/data_stream/antibot/agent/stream/cel.yml.hbs`
- `packages/checkpoint_harmony_endpoint/data_stream/antimalware/agent/stream/cel.yml.hbs`
- `packages/checkpoint_harmony_endpoint/data_stream/forensics/agent/stream/cel.yml.hbs`
- `packages/checkpoint_harmony_endpoint/data_stream/threatextraction/agent/stream/cel.yml.hbs`
- `packages/checkpoint_harmony_endpoint/data_stream/threatemulation/agent/stream/cel.yml.hbs`
- `packages/checkpoint_harmony_endpoint/data_stream/urlfiltering/agent/stream/cel.yml.hbs`
- `packages/checkpoint_harmony_endpoint/data_stream/zerophishing/agent/stream/cel.yml.hbs`
**Location (example stream):**
- `packages/checkpoint_harmony_endpoint/data_stream/antimalware/agent/stream/cel.yml.hbs:69-75`
- `packages/checkpoint_harmony_endpoint/data_stream/antimalware/agent/stream/cel.yml.hbs:170-181`
**Evidence (example stream):**
- New query window starts from cursor `next_startTime`:
- `startTime: state.?cursor.next_startTime.orValue(...)` (`:69-71`)
- On empty completion (`body.data.state == "Done"`), cursor is set to prior start:
- `"next_startTime": state.cursor.current_startTime` (`:180`)
**Runtime failure trace:**
1. Poll N starts timeframe `[current_startTime=S, current_endTime=E]`.
2. API returns `Done` (empty for that timeframe).
3. Cursor update writes `next_startTime = S` instead of advancing to `E`.
4. Poll N+1 builds timeframe from `startTime=S` again, with a newer end time.
5. This repeats indefinitely for periods with no events, re-querying the same historical start boundary and repeatedly widening the queried window.
**What is wrong:**
The empty-result branch does not advance the cursor watermark. For a finished empty window, the next poll should start at the previous window end.
**Why it matters:**
This causes sustained redundant API work and growing query windows during quiet periods, which can hammer the API and consume quota/compute unnecessarily. It also increases risk of delayed ingestion under load.
**Suggested fix:**
In the `Done` branch for all affected templates, advance the cursor to the completed window end:
- Replace `"next_startTime": state.cursor.current_startTime`
- With `"next_startTime": state.cursor.current_endTime`
(Keep retry/error branches (`Canceled`/error responses) unchanged where replaying the same window is intentional.)
## Investigated and found correct
1. `packages/box_events/data_stream/events/agent/stream/httpjson.yml.hbs`
- Pagination stops when `next_stream_position` is absent or page size is below `limit` (`response.pagination` at `:34-39`), so legal terminal responses do not loop forever.
2. `packages/atlassian_jira/data_stream/audit/agent/stream/httpjson.yml.hbs`
- Pagination offset/next-link setters use fail-safe empty template behavior (`:49-60` and server variant `:87-92`), terminating when no further page exists.
3. `packages/google_workspace/data_stream/admin/agent/stream/httpjson.yml.hbs`
- `pageToken` is only set when `nextPageToken` exists (`:47-55`), so missing token cleanly terminates pagination.
> [!WARNING]
>
> ⚠️ Firewall blocked 2 domains
>
> The following domains were blocked by the firewall during workflow execution:
>
> - `developer.box.com`
> - `docscortex.paloaltonetworks.com`
>
> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "developer.box.com"
> - "docscortex.paloaltonetworks.com"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>
> [!NOTE]
>
> 🔒 Integrity filter blocked 1 item
>
> The following item were blocked because they don't meet the GitHub integrity level.
>
> - [#18799](https://github.com/elastic/integrations/issues/18799) `search_issues`: has lower integrity than agent requires. The agent cannot read data with integrity below "approved".
>
> To allow these resources, lower `min-integrity` in your GitHub frontmatter:
>
> ```yaml
> tools:
> github:
> min-integrity: approved # merged | approved | unapproved | none
> ```
>
>
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: httpjson and CEL Pagination and Cursor Integrity](https://github.com/elastic/integrations/actions/runs/26281505932)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on May 29, 2026, 10:19 AM UTC
Contributor guide
Assessment
This issue has not been assessed yet.