[Filebeat] http_endpoint input should not report DEGRADED on client validation failures
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
#### Description
The `http_endpoint` input (health reporting added in PR #44310 per #44281) currently calls `status.Degraded` when a client request fails validation (e.g., wrong HTTP method, failed auth). The same `status.Degraded` mechanism is used for these client validation failures and component level failures.
Also, Fleet uses worst-case aggregation across components, a single rejected request can mark the entire agent as "Unhealthy" in Fleet UI.
The input stays DEGRADED until the next valid request arrives. There is no background health check or periodic reset. This is particularly troublesome for sporadic webhook sources that may go hours/days between events, a single stray GET from a health check probe or network scanner leaves the agent showing Unhealthy the entire time.
Would we be able to improve this behavior such that client validation failures do not affect the input's health status. Rejecting an invalid request is expected, healthy behavior, and not a sign of degradation. Logging the rejection rather than changing health status might be a better fit.
#### Background
The original spec in #44281 only considered infrastructure level conditions like sustained back pressure as potential DEGRADED triggers. Client validation failures were not discussed. The implementation in PR #44310 went beyond this by marking all validation failures as DEGRADED.
#### Current behavior
1. Deploy an Elastic Agent with an `http_endpoint` integration (default config accepts POST only)
2. Send one valid POST. Agent shows `Healthy` in Fleet
3. Send a GET request: `curl http://host:port/path`
4. Agent transitions to `DEGRADED` (HTTP 405)
5. Agent stays DEGRADED until the next valid POST arrives
From a real agent diagnostic bundle, this cycle is visible in the event logs:
```
07:25:54Z Unit state changed (HEALTHY->DEGRADED): request did not validate: only POST requests are allowed
07:29:04Z Unit state changed (DEGRADED->HEALTHY): Healthy ← triggered by a valid POST ~4 mins later
```
#### Affected validation paths
All of these trigger `status.Degraded` in `handler.go`:
- Wrong HTTP method (405)
- Failed basic auth (401)
- Invalid secret header (401)
- Wrong content-type (415)
- Missing, malformed, or mismatched HMAC signature (401)
- Malformed JSON body
- CRC validation failure
- Invalid query parameters
Thank you for considering this. Happy to provide additional detail, diagnostics, or help test a fix.
Contributor guide
Research direction
Start in handler.go and trace the listed http_endpoint validation paths that call status.Degraded. Check the existing health-reporting behavior around rejected requests; done means invalid client requests are still rejected and logged without changing input health, while component-level failures can still report degradation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100