influxdata / influxdata/telegraf

Add Elastic Common Schema (ECS) serializer

Open
#19,187 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

**Use Case**

Elastic Common Schema (ECS) defines how a document's fields are named and nested. Today the only way to get ECS-shaped output from Telegraf is the dedicated elasticsearch output. That output is tied to the Elasticsearch client and its transport. Version checks, node sniffing, index-template management, and _bulk framing all live in one plugin. You cannot produce ECS documents and send them through the generic http output, for example to an ingest gateway in front of Elasticsearch. You also cannot reuse that output's transport options, such as auth, TLS, proxy, and gzip.

Two earlier efforts stalled on this coupling. #19134 was withdrawn because the official Elasticsearch client requires _bulk in the request path. #12537 stalled on mapping arbitrary fields to ECS. A serializer is the better place for this work. ECS describes only the document format, so it belongs in a serializer. The _bulk framing and index routing belong in the output. The change is additive. It does not touch the existing elasticsearch output and adds no new Elasticsearch client dependency.

**Expected behavior**

A new elasticcommonschema serializer (under plugins/serializers/) builds an ECS-compliant JSON document from each metric:

- @timestamp from the metric time, plus ecs.version and event.dataset (from the measurement name)
- tags as top-level fields, with dotted keys expanded into nested objects (for example, orchestrator.cluster.name -> {"orchestrator":{"cluster":{"name":...}}})
- metric fields nested under the measurement name

Serialize emits one document. SerializeBatch emits an NDJSON stream. The serializer does not interact with Elasticsearch and emits no _bulk framing. The _bulk framing stays in the output. Pair the serializer with outputs.http to send ECS documents anywhere, or with an output that handles the _bulk protocol. The configuration is small: an ECS schema version and NaN/Inf float handling.

**Actual behavior**

No ECS serializer exists. ECS-shaped output is only available through the elasticsearch output, and you cannot reuse it with outputs.http or any other output.

**Additional info**

We have a working implementation running in a fork. We are happy to open a PR.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the existing serializers under plugins/serializers/, then compare the elasticsearch output with outputs.http to understand the separation between document serialization and _bulk framing. The work is done when a new elasticcommonschema serializer emits the specified ECS JSON and NDJSON batch documents with the stated configuration options, without adding Elasticsearch client behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.