[streaming-input-resilience] High-severity cursor skip risk in httpjson chain pagination error path
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
## Findings
### 1. `httpjson` chain-pagination advances shared cursor before chain success and swallows transient chain errors
**Priority:** P1 (high)
**Location**
- `x-pack/filebeat/input/httpjson/request.go:625-630`
- `x-pack/filebeat/input/httpjson/request.go:631-637`
- `x-pack/filebeat/input/httpjson/request.go:734-739`
- `x-pack/filebeat/input/httpjson/transform.go:59-67`
**Evidence**
- Cursor is advanced before chain processing succeeds:
- `p.trCtx.updateLastEvent(msg)`
- `p.trCtx.updateCursor()`
- Then chain work is attempted:
- `n, err := p.req.processChainPaginationEvents(...)`
- Transient chain failures are only logged and dropped:
- `if err != nil { ... p.req.log.Errorf("error processing chain event: %v", err); return }`
- Chain fetch path returns real transient/network errors:
- `httpResp, err = rf.collectResponse(...)`
- `return -1, err`
- Transform contexts share the same cursor pointer across clones:
- `newCtx.cursor = ctx.cursor`
**What is wrong**
In the root-pagination + chain flow, the shared cursor can be moved forward for a page/event **before** the chain request(s) for that page succeed. If a chain HTTP call fails transiently, the error is not propagated; processing continues to later root events. Later successful publications can persist a cursor that has moved past the failed chain page.
**Why it matters**
This creates a production-reachable silent data-loss mode under normal transient conditions (temporary 5xx, connection reset, or timeout in chain endpoints): affected chain data can be skipped on subsequent polls/restarts.
**Suggested fix direction**
1. Do not call `updateCursor()` in `chainProcessor.handleEvent` until `processChainPaginationEvents` succeeds.
2. Propagate chain-processing errors to the interval/request level so retries occur from the last safe cursor instead of continuing silently.
3. Add a regression test in `x-pack/filebeat/input/httpjson/request_test.go` that simulates:
- root pagination with chain enabled,
- first chain request failing transiently,
- later events succeeding,
- assertion that persisted cursor never advances past the failed chain page.
## Priority ranking
1. **P1:** Chain-pagination cursor advancement on swallowed transient chain errors (finding above).
## Inputs/error paths reviewed and found safe in this pass
- `x-pack/filebeat/input/streaming/crowdstrike.go:311-337` refresh goroutine lifetime issue is already tracked in open `#50564`.
- `x-pack/filebeat/input/streaming/websocket.go:172,285` token-expiry immediate-trigger loop is already tracked in open `#50446`.
- `x-pack/filebeat/input/streaming/websocket.go:268-273` transient token refresh hard-stop is already tracked in open `#49612`.
- `x-pack/filebeat/input/httpjson/response.go:62-87` invalid response-shape handling degrades/skips rather than publishing non-object events.
## Suggested actions
- [ ] Delay cursor advancement in chain-pagination path until chain success.
- [ ] Return chain errors to retry loop instead of logging-and-continue.
- [ ] Add a focused regression test for transient chain failure + cursor persistence behavior.
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Streaming Input Error Path Resilience](https://github.com/elastic/beats/actions/runs/25911627407)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on May 22, 2026, 10:07 AM UTC
Contributor guide
Assessment
This issue has not been assessed yet.