[metricbeatreceiver] Transient ES connection failure at startup causes permanent error with no recovery
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 364
Description
## Description
The `metricbeatreceiver` does not retry when its initial Elasticsearch connection fails. A single transient `connection refused` at startup (e.g. ES not yet accepting connections) causes the receiver to enter a permanent error state with no recovery path. The collector process stays alive but the receiver is dead and reports `StatusPermanentError` to `healthcheckv2` indefinitely.
This was observed in a Kubernetes environment where the EDOT collector pod started ~2 seconds after ES was reported as ready, but the ES service endpoint was not yet accepting TCP connections. See [elastic/cloud-on-k8s#9165](https://github.com/elastic/cloud-on-k8s/issues/9165) for the full e2e test failure context.
## Error from collector logs
```
error starting metricbeat receiver:
beat receiver run error: failed to register Cloud Connected cluster:
failed to load cluster info: failed to send request:
Get "https://es-host:9200/":
dial tcp :9200: connect: connection refused
```
After this single error, the collector logged `No non-zero metrics` every 30s for 15 minutes until the pod was killed. The `healthcheckv2` extension returned HTTP 500 on every readiness probe.
## Root Cause
The failure chain involves no retry at any layer:
1. **`x-pack/metricbeat/module/autoops_es/metricset/metricset.go`** — `newAutoOpsMetricSet` calls `maybeRegisterCloudConnectedCluster` during module creation. If the ES connection fails, the error propagates immediately.
2. **`x-pack/metricbeat/module/autoops_es/metricset/register.go`** — `maybeRegisterCloudConnectedCluster` calls `getClusterInfo(m)` which makes a single HTTP request to `GET /`. No retry on transient errors.
3. **`metricbeat/beater/metricbeat.go`** — `Metricbeat.Run()` calls `factory.Create()` for each module config. If creation fails, `Run()` returns the error immediately.
4. **`x-pack/libbeat/cmd/instance/receiver.go`** — `BeatReceiver.Start()` calls `beater.Run()`. On failure, it reports `status.Failed` (which maps to `StatusPermanentError`) and returns the error.
5. **`x-pack/metricbeat/mbreceiver/receiver.go`** — `metricbeatReceiver.Start()` runs `BeatReceiver.Start(host)` in a goroutine. If it fails, the error is logged and the goroutine exits. No retry. The `Start()` method itself returns `nil` to the OTel framework, so the collector stays alive while the receiver is permanently dead.
## Expected Behavior
A transient connection failure (e.g. `connection refused`, DNS resolution failure, TCP timeout) during the initial Cloud Connected cluster registration should be retried with backoff, similar to the existing retry logic used for the HTTP API server in `NewBeatReceiver` (which already uses `backoff.NewRetryer`).
## Related
- #47848 / #47936 — Fixed status reporting so that `BeatReceiver.Start()` properly reports `status.Failed` when `beater.Run()` fails. This made the failure visible via health checks but did not add retry logic.
- [elastic/cloud-on-k8s#9165](https://github.com/elastic/cloud-on-k8s/issues/9165) — Downstream e2e test failure triggered by this bug.
Contributor guide
Assessment
This issue has not been assessed yet.