influxdata / influxdata/telegraf
dedup: Expand dedup functionality
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
Many metrics contain fields that always change (e.g., timestamps, device uptimes) but still need to be included in output for downstream processing. The current dedup logic compares using these changing fields when determining whether to drop a metric, which causes excessive “false change” detections.
I want the ability to:
- Exclude specific fields/attributes from comparison while still preserving them in the output metric, like a metric count that just happens to come from a device I'm collecting from. A config like ignore_keys would be nice.
- Compare only a certain object, or certain multiple objects, within a metric: like the fields object for instance for changes, ignoring tags or other attributes for dedup purposes.
- Choose the key/compound key that ultimately ends up identifying the metric (and possibly set fallbacks)
- Forward or drop unidentifiable metrics with no key or clear source (name attribute might not exist or something)
This would allow me to reduce unnecessary duplicate metrics while maintaining the completeness of the output for systems like InfluxDB and Grafana. If I plan to pull every nanosecond for some reason but I also cannot handle loading that much data to my DB, I can get the benefits of collection granularity and event driven change detection by only receiving data when there are detected changes.
### Expected behavior
Dedup should allow ignoring specific keys while still outputting them, restricting comparisons to certain objects (e.g., only fields), defining custom or compound identifying keys for metrics (with fallbacks), and optionally dropping unidentifiable metrics. This lets me simply filter noisy sources without losing important data.
These features were useful to me when I built them for my use-case.
### Actual behavior
Dedup currently compares all fields and tags, with no way to ignore always-changing keys, limit comparisons to certain objects, or define custom metric keys. This causes frequent false changes and prevents efficient filtering.
### Additional info
I have used Telegraf’s dedup processor to reduce unnecessary metric updates in high-volume network telemetry pipelines but transitioned to DIY because it lacks the customization necessary to filter repetitive metrics. I have some suggestions about how dedup can be improved for chatty data sources.
Dedup could be the go-to plugin to "reduce data output significantly without actually dropping any data (that often...)". That would be really helpful. Current dedup just isn't there yet - it could be more flexible in my opinion.
Contributor guide
Research direction
The payload names Telegraf’s dedup processor but no source files or tests. Start by locating that processor and its existing configuration and tests, then clarify the comparison scope, identifying-key fallbacks, and handling of unidentifiable metrics. Done means the requested dedup behaviors are specified, configurable, preserved in output, and covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100