elastic / elastic/elastic-agent

[bug] OOMKill - Prometheus Reader Accumulation

Open
#14,734 4 comments 1 reaction 0 assignees View on GitHub
bug Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
275
Forks
264
Avg merge
1d 23h
Merged PRs (30d)
312

Description

## Summary

Elastic Agent in k8s pods running versions 9.3.2–9.4.1 experienced unbounded memory growth and OOMKill.

Per Claude analysis, the root cause is a bug in `internal/pkg/otel/manager/execution_subprocess.go` where `addCollectorMetricsReader` mutated the shared `m.mergedCollectorCfg` pointer in-place on every OTel collector subprocess start. A separate pre-existing misconfiguration (no OTel pipelines/receivers) put the collector into a continuous crash-restart loop, giving the memory bug thousands of opportunities to compound.

## Timeline

| Time | Event |
|---|---|
| 2026-02-17 | Bug introduced on `main` (PR #12809) — `addCollectorMetricsReader` added inside `startCollector` |
| 2026-03-31 | Fix merged to `main` (PR #13250) — config modification moved out of `startCollector` |
| 2026-04-10 | `main` bumped to 9.5.0; 9.4.x release branch cut **before** fix was cherry-picked |
| 2026-05-18 | Crash loop detected on affected pods; 21,000+ restarts logged by May 20 |
| 2026-05-20 | Diagnostics captured; `otel-merged.yaml` at 6 MB (22,184 duplicate readers), live heap at 201 MB |

## Root Cause

### Precondition: OTel collector in an infinite crash loop

A policy misconfiguration (dynamic CEL inputs routed to the process runtime via `dynamic_inputs: process`) caused the OTel manager to receive no components. `buildMergedConfig` still started the collector because the Fleet policy's `otel.yaml` was non-nil (contained a `headers_setter` extension), but the merged config had no `receivers` or `service.pipelines`. The collector failed immediately on every start:

```
invalid configuration: no receiver configuration specified in config
service::pipelines: service must have at least one pipeline
```

This was known and confirmed separately. It is the **precondition** that amplified the memory bug.

### The memory bug: mutation of shared config pointer

In versions prior to commit bda911f8a0, `startCollector` in `execution_subprocess.go` defined a `configModifier` closure that was applied directly to the `cfg *confmap.Conf` argument (which is `m.mergedCollectorCfg`, a shared pointer):

```go
// OLD CODE (buggy) — execution_subprocess.go
var configModifier ConfigModifier = func(cfg *confmap.Conf) error {
injectHealthCheckV2Extension(cfg, ...)
addCollectorMetricsReader(cfg, r.collectorMetricsPort) // mutates cfg in-place
return nil
}
prepareAndSerializeConfig(cfg, configModifier) // calls configModifier(cfg)
```

`addCollectorMetricsReader` reads the existing `service::telemetry::metrics::readers` list from `cfg`, appends one new prometheus pull reader, and merges it back via `confmap.Conf.Merge` — which mutates the receiver in-place via koanf. Because `cfg` **is** `m.mergedCollectorCfg`, each call permanently grew the shared config by one reader.

The same modifier was also stored in `procHandle.configModifier` and applied by `UpdateConfig` on every live config reload.

**Effect per restart cycle (~18 s):**

```
m.mergedCollectorCfg.readers: N → N+1
```

After 22,184 restart cycles:
- `service.telemetry.metrics.readers`: 22,184 identical prometheus pull entries
- `otel-merged.yaml`: 6 MB (normal: ~21 KB)
- Each restart serialized the full config via `yaml.Marshal` → ~200 MB live heap
- Cumulative allocations: ~20.4 TB

### Fix

Commit bda911f8a0 ("Move otel config modification completely into the otel manager", 2026-03-31) removed the `configModifier` pattern entirely. `addCollectorMetricsReader` was moved into `buildMergedConfig`, which always starts from a fresh `confmap.New()` — eliminating the shared-pointer mutation.

This fix is present on `main` (≥ 9.5.0) but was **not backported** to the 9.3.x or 9.4.x release branches.

## Affected Versions

The bug was introduced by PR #12809 (2026-02-17) and backported to 9.3.2. All versions from 9.3.2 through 9.4.1 are affected.

Versions ≤ 9.3.1 predate the introduction of `addCollectorMetricsReader` and are not affected. Versions ≥ 9.5.0 include the fix.

## Required Actions

1. Backport bda911f8a0 to the 9.3.x and 9.4.x release branches.
2. **Investigate the pipeline misconfiguration** (Bug 1) that put the collector in a crash loop in the first place. Without the crash loop, the reader accumulation would be negligible in practice.

Contributor guide

Open the contributing guide

Research direction

Start with internal/pkg/otel/manager/execution_subprocess.go and compare the affected release branches with commit bda911f8a0, which moved addCollectorMetricsReader into buildMergedConfig. Backport the fix to the 9.3.x and 9.4.x branches, then investigate the dynamic_inputs: process pipeline misconfiguration that caused the collector crash loop. Done means the fix is applied on both branches and the precondition is understood or separately addressed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.