elastic / elastic/beats

[metricbeatreceiver] Transient ES connection failure at startup causes permanent error with no recovery

Open
#49,099 1 comment 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

## Description

The `metricbeatreceiver` does not retry when its initial Elasticsearch connection fails. A single transient `connection refused` at startup (e.g. ES not yet accepting connections) causes the receiver to enter a permanent error state with no recovery path. The collector process stays alive but the receiver is dead and reports `StatusPermanentError` to `healthcheckv2` indefinitely.

This was observed in a Kubernetes environment where the EDOT collector pod started ~2 seconds after ES was reported as ready, but the ES service endpoint was not yet accepting TCP connections. See [elastic/cloud-on-k8s#9165](https://github.com/elastic/cloud-on-k8s/issues/9165) for the full e2e test failure context.

## Error from collector logs

```
error starting metricbeat receiver:
beat receiver run error: failed to register Cloud Connected cluster:
failed to load cluster info: failed to send request:
Get "https://es-host:9200/":
dial tcp :9200: connect: connection refused
```

After this single error, the collector logged `No non-zero metrics` every 30s for 15 minutes until the pod was killed. The `healthcheckv2` extension returned HTTP 500 on every readiness probe.

## Root Cause

The failure chain involves no retry at any layer:

1. **`x-pack/metricbeat/module/autoops_es/metricset/metricset.go`** — `newAutoOpsMetricSet` calls `maybeRegisterCloudConnectedCluster` during module creation. If the ES connection fails, the error propagates immediately.

2. **`x-pack/metricbeat/module/autoops_es/metricset/register.go`** — `maybeRegisterCloudConnectedCluster` calls `getClusterInfo(m)` which makes a single HTTP request to `GET /`. No retry on transient errors.

3. **`metricbeat/beater/metricbeat.go`** — `Metricbeat.Run()` calls `factory.Create()` for each module config. If creation fails, `Run()` returns the error immediately.

4. **`x-pack/libbeat/cmd/instance/receiver.go`** — `BeatReceiver.Start()` calls `beater.Run()`. On failure, it reports `status.Failed` (which maps to `StatusPermanentError`) and returns the error.

5. **`x-pack/metricbeat/mbreceiver/receiver.go`** — `metricbeatReceiver.Start()` runs `BeatReceiver.Start(host)` in a goroutine. If it fails, the error is logged and the goroutine exits. No retry. The `Start()` method itself returns `nil` to the OTel framework, so the collector stays alive while the receiver is permanently dead.

## Expected Behavior

A transient connection failure (e.g. `connection refused`, DNS resolution failure, TCP timeout) during the initial Cloud Connected cluster registration should be retried with backoff, similar to the existing retry logic used for the HTTP API server in `NewBeatReceiver` (which already uses `backoff.NewRetryer`).

## Related

- #47848 / #47936 — Fixed status reporting so that `BeatReceiver.Start()` properly reports `status.Failed` when `beater.Run()` fails. This made the failure visible via health checks but did not add retry logic.
- [elastic/cloud-on-k8s#9165](https://github.com/elastic/cloud-on-k8s/issues/9165) — Downstream e2e test failure triggered by this bug.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.