influxdata / influxdata/telegraf

Feature request: Add Avro serialization for Kafka keys and values

Open
#19,449 2 comments 0 reactions 0 assignees View on GitHub
feature request help wanted size/m
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

Our current telemetry pipeline is:

Telegraf -> Apache NiFi -> Kafka

Telegraf collects and enriches the metrics. Apache NiFi then serializes
the Kafka message key and value using Avro and publishes the messages to
Kafka.

The source systems and Kafka are located in different network zones.
Using Avro is important because it reduces the amount of data transferred
between these zones.

We want to remove Apache NiFi and use Telegraf as the only component in
the pipeline:

Telegraf -> Kafka

This would allow Telegraf to collect, enrich, serialize, and publish the
metrics directly to Kafka.

### Expected behavior

The `outputs.kafka` plugin should support Avro serialization for both:

- the Kafka message key;
- the Kafka message value.

The serialized key and value should use binary Avro encoding and the
Confluent wire format.

The key and value should preferably support separate Avro schemas and
schema IDs, since they may use different schemas.

The Avro schema is defined once and reused for serialization.

A possible configuration could look like this:

[[outputs.kafka]]
brokers = ["kafka:9092"]
topic = "metrics"

key_data_format = "avro"
data_format = "avro"

schema_registry_url = "https://schema-registry.example.com"

key_schema_id = 10
value_schema_id = 11

# Optional TLS configuration
# tls_ca = "/etc/telegraf/certs/ca.pem"
# insecure_skip_verify = false

Telegraf should:

1. Read the configured schema ID for the Kafka key and value.
2. Check whether the corresponding schema is available in the local
in-memory cache.
3. Fetch the schema from Schema Registry over HTTPS on a cache miss.
4. Store the retrieved schema in the cache.
5. Reuse the cached schema for subsequent messages.
6. Serialize the Kafka key and value using the corresponding schemas.
7. Publish the resulting Confluent-compatible payload to Kafka.

The exact configuration names are illustrative and can be adjusted to
match the existing Telegraf conventions.

### Actual behavior

Telegraf currently supports Avro as an input data format, but it does
not support Avro serialization for Kafka output messages.

In particular, `outputs.kafka` cannot currently:

- serialize the Kafka message key as Avro;
- serialize the Kafka message value as Avro;
- produce Kafka key and value payloads in the Confluent wire format.

As a result, the current pipeline cannot be migrated completely to:

Telegraf -> Kafka

Using JSON or another text-based format instead of Avro would increase
the amount of data transferred between the network zones.

### Additional info

The main goal is to remove Apache NiFi and reduce the operational and
network overhead of the current pipeline.

A minimal first implementation could support:

- the Kafka output plugin only;
- binary Avro encoding;
- user-provided schemas;
- separately configured key and value schema IDs;
- fetching existing schemas from Schema Registry over HTTPS;
- in-memory schema caching;
- Confluent wire format.

Schema registration, dynamic schema generation, and advanced schema
evolution support could be added later.

This request is related to issue #1630:

https://github.com/influxdata/telegraf/issues/1630

That issue requested Avro support for Kafka producer and consumer. The
current implementation provides Avro parsing, but Avro output
serialization is still required for this use case.

Contributor guide

Open the contributing guide

Research direction

Start by reading the outputs.kafka plugin and Telegraf’s existing Avro input implementation, then review the related issue #1630. Define the smallest implementation around separate key and value schemas, Schema Registry HTTPS fetching, in-memory caching, and Confluent wire-format output. Done means Kafka keys and values can be serialized with configured Avro schema IDs.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.