elastic / elastic/integrations

[httpjson-pagination] PANW Cortex XDR httpjson next_ts cursor can serialize to scientific notation

Open
#19,392 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Findings

### 1. `panw_cortex_xdr` httpjson streams store millisecond cursor values as untyped JSON numbers, then reuse them directly in request filters

**Severity:** High

**Locations**
- `packages/panw_cortex_xdr/data_stream/incidents/agent/stream/httpjson.yml.hbs:47-53,80-82`
- `packages/panw_cortex_xdr/data_stream/alerts/agent/stream/httpjson.yml.hbs:47-53,80-82`
- Numeric source examples in package docs/samples:
- `packages/panw_cortex_xdr/data_stream/incidents/sample_event.json:33` (`"modification_time":1691976000230`)
- `packages/panw_cortex_xdr/data_stream/alerts/sample_event.json:34` (`"local_insert_ts":1673372647792`)

**Evidence**
The templates write cursor values from event fields and inject them directly into JSON request bodies without explicit integer/string normalization:

```yaml
# incidents
"value": [[ .cursor.next_ts ]]
...
cursor:
next_ts:
value: '[[if index .last_event "modification_time"]][[.last_event.modification_time]][[end]]'

# alerts
"value": [[ .cursor.next_ts ]]
...
cursor:
next_ts:
value: '[[if index .last_event "local_insert_ts"]][[.last_event.local_insert_ts]][[end]]'
```

`httpjson` cursor state is JSON-serialized. JSON numeric values are decoded as `float64` on readback. Reproduction (Go default formatting used by templating):

```go
json.Unmarshal([]byte(`{"next_ts":1709061312345}`), &m)
// type/value: float64 1.709061312345e+12
fmt.Sprint(m["next_ts"])
// 1.709061312345e+12
```

**What is wrong**
A millisecond cursor that was originally an integer can be rendered as scientific notation when reused from `.cursor.next_ts`.

**Why it matters**
This value is used as the incremental boundary (`modification_time` / `server_creation_time`) for polling. Sending exponent-form timestamps can cause malformed/changed filter semantics at the API boundary, which can stop incremental collection or shift boundaries (duplicates/missed events).

**Suggested fix**
Normalize cursor type before persistence and before interpolation:
1. Store `next_ts` as canonical integer/string digits.
2. In request transform, enforce numeric formatting explicitly (for example, `toInt` before embedding).
3. Apply the same fix to both incidents and alerts httpjson templates to avoid divergence.

## Investigated and found correct

- `packages/panw_cortex_xdr/data_stream/alerts/agent/stream/cel.yml.hbs:84-96` uses typed CEL math (`int(...)`) for cursor progression, avoiding float/scientific-notation drift.
- Existing Box scientific-notation issue is already tracked (`#18759`) and not duplicated here.

## Suggested Actions

- [ ] Normalize `next_ts` handling in PANW Cortex XDR httpjson incidents template.
- [ ] Apply the same normalization in PANW Cortex XDR httpjson alerts template.
- [ ] Add a regression test with a 13-digit millisecond cursor round-trip to assert non-scientific request output.

> [!WARNING]
>
> ⚠️ Firewall blocked 1 domain
>
> The following domain was blocked by the firewall during workflow execution:
>
> - `docs.aws.amazon.com`
>
> To allow these domains, add them to the `network.allowed` list in your workflow frontmatter:
>
> ```yaml
> network:
> allowed:
> - defaults
> - "docs.aws.amazon.com"
> ```
>
> See [Network Configuration](https://github.github.com/gh-aw/reference/network/) for more information.
>
>

---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: httpjson and CEL Pagination and Cursor Integrity](https://github.com/elastic/integrations/actions/runs/27009344529)

Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on Jun 12, 2026, 10:34 AM UTC

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.