elastic / elastic/beats

Introduced failureThreshold causes agents to become unhealthy

Open
#42,672 6 comments 0 reactions 0 assignees View on GitHub
bug Stalled Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Due to the introduced failureThreshold (https://github.com/elastic/beats/pull/41570 & https://github.com/elastic/elastic-agent/pull/5999) our agents now become unhealthy duo to runtime errors in Kubernetes environment.

Expected runtime errors include the following:

`[elastic_agent][warn] Unit state changed filestream-default-filestream-container-logs-3b56171a-c46b-475e-9a80-02708c67ce0c-kubernetes-755830fc-44b9-4b02-9333-bb2653b9ae47.podxyz (HEALTHY->DEGRADED): error while reading from source: context cancel`

This might be caused by Kubernetes removing container log-files from containers, that dont necessarily exist anymore.

`[elastic_agent][warn] Unit state changed prometheus/metrics-default-prometheus/metrics-prometheus-cfa93471-3f87-4d04-babb-ef2a62a85cd4-kubernetes-3009cd60-7f0c-44d8-8b85-4257683395ef (HEALTHY->DEGRADED): Error fetching data for metricset prometheus.collector: unable to decode response from prometheus endpoint: error making http request: Get "http://255.255.255.255:8888/metrics": dial tcp 255.255.255.255:8888: connect: connection refused`

This can be caused by prometheus-integration trying to scrape completed jobs and therefore completed pods (see https://github.com/elastic/elastic-agent/issues/6154 for reference).

For confirmed bugs, please report:
- Version: 8.16.X-8.17.X
- Operating System: RHEL 8.9

Contributor guide

Open the contributing guide

Research direction

Start with the failureThreshold changes in the linked Beats and Elastic Agent pull requests, then compare the Kubernetes filestream and Prometheus runtime warnings with the agent health transition. Done means the confirmed failure path and a reproducible fix are established for the reported 8.16.x–8.17.x RHEL 8.9 environments.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, prometheus
Domain
devops, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.