elastic / elastic/beats

Metricbeat publishes error-only events (no metric data) to metrics datastreams

Open
#48,920 6 comments 0 reactions 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

## Summary

When a metricset's `Fetch` method returns an error, the Metricbeat framework publishes an event to the output that contains **only** `error.message` and metadata — **no actual metric data**. This applies to **all** metricsets, as the framework itself is responsible for sending these error events.

With the adoption of TSDB-backed data streams (TSDS), this behavior can be considered problematic: these error-only events are not metrics, yet they are written to metrics datastreams. In TSDS mode, documents that lack the required dimension fields (e.g. specific metric fields that are mapped as dimensions) are **rejected by Elasticsearch**, possibly causing indexing failures.

It's not an metricbeat exclusive issue. For example filestream's CEL input does that as well. If it's used to collect metrics instead of logs, then it's the same issue. At least the Logstash integration does that.

## How it works today

### The code path

When any metricset's `Fetch` returns a non-nil error, the framework's [`handleFetchError`](https://github.com/elastic/beats/blob/main/metricbeat/mb/module/wrapper.go#L310-L333) calls `reporter.Error(err)`:

[`reporter.Error(err)`](https://github.com/elastic/beats/blob/main/metricbeat/mb/module/wrapper.go#L377) creates an `mb.Event` with **only** the `Error` field set — no `MetricSetFields`, no `ModuleFields`, no `RootFields`:

This event goes through the normal pipeline and [`BeatEvent()`](https://github.com/elastic/beats/blob/main/metricbeat/mb/event.go#L104-L108) converts it into:

```go
if e.Error != nil {
b.Fields["error"] = mapstr.M{
"message": e.Error.Error(),
}
}
```

This applies to **every** metricset that implements `ReportingMetricSetV2Error` or `ReportingMetricSetV2WithContext`, which is effectively all of them. The error event is always published to the same data stream as the normal metrics.

Additionally, some metricsets also call `reporter.Error()` directly within their `Fetch` method for per-item errors (e.g. `redis/key`, `mongodb/dbstats`, `mongodb/collstats`, `prometheus/query`, `nats/connection`, etc.), producing the same kind of error-only event inline.

### Fields on the error event

The error event goes through two enrichment stages:

1. **Metricbeat layer** (`reporterV2.Event()` + `AddMetricSetInfo` + `BeatEvent()`) — adds metricset metadata and the error.
2. **libbeat processing pipeline** (`libbeat/publisher/processing/default.go`) — adds builtin fields configured at startup via [`WithAgentMeta()`](https://github.com/elastic/beats/blob/main/libbeat/publisher/processing/default.go#L161-L177), `WithHost`, and `WithECS`.

The resulting document contains:

| Field | Value |
|---|---|
| `@timestamp` | fetch start time |
| `error.message` | the error string |
| `service.type` | module name |
| `service.address` | host URI (if configured) |
| `event.dataset` | `.` |
| `event.module` | module name |
| `event.duration` | fetch round-trip time |
| `metricset.name` | metricset name |
| `metricset.period` | configured period |
| `agent.id` | agent UUID |
| `agent.name` | agent hostname/name |
| `agent.type` | `metricbeat` |
| `agent.version` | beat version |
| `agent.ephemeral_id` | ephemeral UUID |
| `host.name` | hostname |
| `ecs.version` | ECS version |

Notably **absent**: any `..*` metric fields, and any fields that the data stream mapping expects as dimensions.

## The problem with TSDB / TSDS

### These events are not metrics

The error-only event contains no metric data — it is effectively a **log entry** (an error message with metadata) being written to a **metrics** data stream. This is a semantic mismatch: metrics data streams should contain metrics.

### TSDS dimension requirements cause document rejection

In TSDB-backed data streams, every document must include at least one field mapped as `time_series_dimension`. These dimension. Not all metricsets and integrations fetch data from the host, so
it might not make sense to have agent or host metadata as dimensions. If no
dimension is defined, the event is rejected.

### Even when not rejected, it pollutes the data stream

In non-TSDS data streams (or if the dimensions happen to match), the error-only document still gets indexed. But it contains no metric values, which can:
- Inflate document counts without adding useful metric data
- Break dashboards and aggregations that don't explicitly filter out `error.message:*`
- Create gaps in time series visualizations (a document exists at that timestamp but with no metric values)
- The error events might be aggregated under the same `_tsid` when they belong to different entities

## Open questions

1. **Should we stop publishing error-only events to the metrics data stream?** Errors are already surfaced via Metricbeat logs and module health status (`Degraded`/`Running` via the Agent control protocol). Is the in-band error event actually needed?

2. **If we still want in-band error signaling, where should it go?** Options include:
- A separate data stream for collection errors
- Only logging + status reporting (no event at all)
- An event with a distinct `event.kind` (e.g. `event` or `alert` instead of the implicit metric kind) routed elsewhere

Contributor guide

Open the contributing guide

Research direction

Start by tracing metricbeat/mb/module/wrapper.go, especially handleFetchError and reporter.Error, then follow event.go's BeatEvent and libbeat/publisher/processing/default.go. Compare framework-generated and inline error-only events, and clarify the intended handling for metrics data streams; done requires an agreed behavior that addresses TSDB rejection and data-stream pollution.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.