Introduced failureThreshold causes agents to become unhealthy
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
Due to the introduced failureThreshold (https://github.com/elastic/beats/pull/41570 & https://github.com/elastic/elastic-agent/pull/5999) our agents now become unhealthy duo to runtime errors in Kubernetes environment.
Expected runtime errors include the following:
`[elastic_agent][warn] Unit state changed filestream-default-filestream-container-logs-3b56171a-c46b-475e-9a80-02708c67ce0c-kubernetes-755830fc-44b9-4b02-9333-bb2653b9ae47.podxyz (HEALTHY->DEGRADED): error while reading from source: context cancel`
This might be caused by Kubernetes removing container log-files from containers, that dont necessarily exist anymore.
`[elastic_agent][warn] Unit state changed prometheus/metrics-default-prometheus/metrics-prometheus-cfa93471-3f87-4d04-babb-ef2a62a85cd4-kubernetes-3009cd60-7f0c-44d8-8b85-4257683395ef (HEALTHY->DEGRADED): Error fetching data for metricset prometheus.collector: unable to decode response from prometheus endpoint: error making http request: Get "http://255.255.255.255:8888/metrics": dial tcp 255.255.255.255:8888: connect: connection refused`
This can be caused by prometheus-integration trying to scrape completed jobs and therefore completed pods (see https://github.com/elastic/elastic-agent/issues/6154 for reference).
For confirmed bugs, please report:
- Version: 8.16.X-8.17.X
- Operating System: RHEL 8.9
Contributor guide
Research direction
Start with the failureThreshold changes in the linked Beats and Elastic Agent pull requests, then compare the Kubernetes filestream and Prometheus runtime warnings with the agent health transition. Done means the confirmed failure path and a reproducible fix are established for the reported 8.16.x–8.17.x RHEL 8.9 environments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes, prometheus
- Domain
- devops, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100