influxdata / influxdata/telegraf
Input plugin: allow parsing of compressed payload (eg: bson, json)
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
With many devices pushing telemetry on a MQTT broker, we'd like to transfer compressed structured data instead of plain JSON text. For example, we could format payload as BSON or MessagePack data, and zlib' or zstd' the binary payload.
Data transfer gains would be huge, sometimes 5x or 10x less bytes.
Maybe the solution already exists and I did not find the right information about this.
### Expected behavior
Telegraf would accept a compression_format = zstd in the plugin configuration, or we could pass payloads through several different plugins to expand it.
Parsing BSON or MessagePack in Json v2 would be great too.
### Actual behavior
Parse JSON, without compression
### Additional info
_No response_
Contributor guide
Research direction
Review Telegraf's input plugin configuration and Json v2 parsing path, then check whether existing plugin chaining can handle decompression. Clarify whether the first milestone is zlib/zstd support, BSON/MessagePack parsing, or both, and define tests for compressed MQTT payloads; done means the selected behavior is accepted consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, json
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100