elastic / elastic/beats

Elastic Agent on K8s - Improve logging on falling behind reading the event queue of K8s

Open
#46,671 1 comment 0 reactions 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

**Describe the enhancement:**

When low on resources, Elastic Agent used to implement K8s observability use cases might log:

```
{"log.level":"error","@timestamp":"2025-09-15T00:31:18.803Z","message":"I0915 00:31:16.372977 43 trace.go:236] Trace[997072019]: \"DeltaFIFO Pop Process\" ID:kube-system/konnectivity-agent-b5787dd87-h4z4k,Depth:12,Reason:slow event handlers blocking the queue (15-Sep-2025 00:31:11.464) (total time: 236ms):","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2025-09-15T00:31:18.810Z","message":"Trace[997072019]: [236.621333ms] [236.621333ms] END","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}
```

The message is quite obscure and reported in several posts online:
- https://discuss.elastic.co/t/metricbeat-crashing-reason-slow-event-handlers-blocking-the-queue/306648/3
- https://discuss.elastic.co/t/elastic-agent-not-shipping-all-logs-from-kubernetes-cluster-errors-in-logs/325620

**Describe a specific use case for the enhancement or feature:**

We should attempt to improve the experience of the user and make those errors actionable, such as:
- If possible, return the state of the unit as `degraded` until it recovers.
- Suggest to increase the resources of the host / container in the logs

FYI @ebeahan

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.