elastic / elastic/integrations

[httpjson-pagination] Box Events cursor can regress to scientific notation and corrupt stream_position

Open
#18,759 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Findings

### 1. Box Events `next_stream_position` cursor can be emitted in scientific notation

**Severity:** High

**Location:**
- `packages/box_events/data_stream/events/agent/stream/httpjson.yml.hbs:20`
- `packages/box_events/data_stream/events/agent/stream/httpjson.yml.hbs:37`
- `packages/box_events/data_stream/events/agent/stream/httpjson.yml.hbs:42`
- Supporting sample payload: `packages/box_events/data_stream/events/_dev/test/pipeline/test-empty.log:1`

**Evidence:**

`httpjson.yml.hbs` directly interpolates cursor/page token values without integer/string normalization:

```yaml
# L20
value: '[[if index .cursor "next_stream_position"]][[.cursor.next_stream_position]][[else]][["0"]][[end]]'

# L37
value: '[[ if and (index .last_response.body "next_stream_position") (ge (len .last_response.body.entries) \{\{limit}}) ]][[.last_response.body.next_stream_position]][[end]]'

# L42
value: '[[.last_response.body.next_stream_position]]'
```

The package also contains a numeric `next_stream_position` sample:

```json
{"chunk_size":0,"entries":[],"next_stream_position":564670747717431}
```

Runtime reproduction of JSON round-trip behavior for this value:
- JSON unmarshal type: `float64`
- Rendered form: `5.64670747717431e+14`

So a numeric cursor value can be persisted and then sent as:

```text
...?stream_position=5.64670747717431e+14
```

instead of the expected decimal token.

**What is wrong:**
The stream assumes `next_stream_position` is always safe to interpolate as-is. If the API returns a numeric token (as seen in package test data), cursor JSON round-trip converts it to `float64`, and later template rendering emits scientific notation.

**Why it matters:**
`stream_position` is the pagination/cursor anchor for exactly-once collection. Scientific notation or precision drift can make the next request start from the wrong position, causing duplicate events, missed events, or stalled pagination.

**Suggested fix:**
Normalize `next_stream_position` before storing/using it in request params, e.g. enforce canonical decimal string/integer formatting in both request transform and cursor assignment (similar to the prior large-offset handling that existed before `bdc4247035`).

## Investigated and found correct

- `packages/m365_defender/data_stream/alert/agent/stream/httpjson.yml.hbs` and `.../incident/...`: cursor field is RFC3339 datetime string (`lastUpdateDateTime`), so no numeric scientific-notation drift path in cursor serialization.
- `packages/ti_eset/data_stream/ip/agent/stream/httpjson.yml.hbs`: cursor is string header/token based, not large numeric cursor state.
- `packages/panw_cortex_xdr/data_stream/incidents/agent/stream/httpjson.yml.hbs`: cursor/time filters are consistently handled as numeric values in request body without string interpolation of JSON-round-tripped cursor tokens.

> [!WARNING]
>
> ⚠️ Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `developer.box.com`
>
> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "developer.box.com"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: httpjson and CEL Pagination and Cursor Integrity](https://github.com/elastic/integrations/actions/runs/25209738540)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on May 8, 2026, 9:47 AM UTC

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.