influxdata / influxdata/telegraf

Input plugin: allow parsing of compressed payload (eg: bson, json)

Open
#16,252 1 comment 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

With many devices pushing telemetry on a MQTT broker, we'd like to transfer compressed structured data instead of plain JSON text. For example, we could format payload as BSON or MessagePack data, and zlib' or zstd' the binary payload.
Data transfer gains would be huge, sometimes 5x or 10x less bytes.

Maybe the solution already exists and I did not find the right information about this.

### Expected behavior

Telegraf would accept a compression_format = zstd in the plugin configuration, or we could pass payloads through several different plugins to expand it.
Parsing BSON or MessagePack in Json v2 would be great too.

### Actual behavior

Parse JSON, without compression

### Additional info

_No response_

Contributor guide

Open the contributing guide

Research direction

Review Telegraf's input plugin configuration and Json v2 parsing path, then check whether existing plugin chaining can handle decompression. Clarify whether the first milestone is zlib/zstd support, BSON/MessagePack parsing, or both, and define tests for compressed MQTT payloads; done means the selected behavior is accepted consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, json
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.