[streaming-input-resilience] httpjson stale rate-limit reset can trigger tight retry loop
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
## Findings
### 1. `httpjson` can spin in an unbounded retry loop on realistic `429` responses
**Priority:** P1 (high)
**Location**
- `x-pack/filebeat/input/httpjson/rate_limiter.go:51-70`
- `x-pack/filebeat/input/httpjson/rate_limiter.go:83-87`
- `x-pack/filebeat/input/httpjson/rate_limiter.go:174-176`
- `x-pack/filebeat/input/httpjson/request.go:267-289`
**Evidence**
- `rateLimiter.execute` loops forever until it sees `200` or rate-limit is "not applied":
- `for { ... if resp.StatusCode == http.StatusOK || !applied { return resp, nil } }` (`rate_limiter.go:51-70`)
- When reset time is in the past (or wait computes non-positive), code returns `limitReached=true` with no wait:
- `if resumeAt == 0 || w <= 0 { ... return limitReached, nil }` (`rate_limiter.go:83-87`)
- `if timeNow().Unix() > resumeAt { return true, 0, nil }` (`rate_limiter.go:174-176`)
- `httpClient.do` calls `c.limiter.execute(...)` before status handling (`request.go:267-289`), so this loop can repeatedly call the endpoint without backoff when `429` keeps coming.
**What is wrong**
A server that keeps returning `429` with a stale/past reset timestamp causes `applyRateLimit` to report "limit reached" but skip sleeping. Because `execute` retries while `applied=true` and status is not `200`, the input can enter an immediate retry loop.
**Why it matters**
This is production-reachable with real APIs (clock skew, stale `X-RateLimit-Reset`, or bad upstream implementations). Impact is high: endpoint hammering, CPU/network churn, and possible ingestion starvation while stuck in the loop.
**Suggested fix direction**
1. In `rateLimiter.execute`, enforce bounded retries/backoff for repeated non-200 responses even when rate-limit metadata is malformed/stale.
2. In `applyRateLimit`, when `limitReached==true` and computed wait is non-positive, apply a minimum delay (or fail the request) instead of immediate retry.
3. Keep context cancellation responsive while waiting.
**Suggested failing test**
Add a focused unit test in `x-pack/filebeat/input/httpjson/rate_limiter_test.go` that simulates repeated `429` responses with reset timestamps in the past and asserts the limiter does **not** perform tight immediate retries (e.g., bounded attempts in a time window or enforced minimum delay).
## Inputs and error paths checked and found safe
- `x-pack/filebeat/input/httpjson/input.go:240-283`: periodic loop logs request errors and keeps running.
- `x-pack/filebeat/input/httpjson/request.go:280-288`: non-2xx responses are converted to `httpError` (not published as normal events).
- `x-pack/filebeat/input/streaming/crowdstrike.go:188-234`: reconnect path uses backoff and preserves session state.
## Already-tracked items intentionally not duplicated
- `#50446`: websocket token-expiry immediate refresh/reconnect loop.
- `#49612`: websocket transient token-refresh hard-stop and other resilience gaps.
- `#50719`: httpjson chain-pagination cursor advancement on chain error path.
---
[What is this?](https://ela.st/github-ai-tools) | [From workflow: Sweeper: Streaming Input Error Path Resilience](https://github.com/elastic/beats/actions/runs/26281431488)
Give us feedback! React with 🚀 if perfect, 👍 if helpful, 👎 if not.
> - [x] expires on May 29, 2026, 10:16 AM UTC
Contributor guide
Assessment
This issue has not been assessed yet.