influxdata / influxdata/telegraf

Configuration setting for creating multiple instances of plugins for parallel processing

Open
#11,707 2 comments 1 reaction 1 assignee Claimed by @srebhan View on GitHub
feature request performance plugin/processor size/l
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

What if telegraf processor plugins had a `parallel instance count` configuration, allowing multiple, instances of a processor to be initialized from the same configuration? Telegraf would partition the metrics evenly between each instance and metrics would flow to the next stage of the pipeline normally.

Background:
My team and I use telegraf to scrape prometheus-style metrics from an endpoint that exposes a rather large number of metrics. We need to do some transformation of these metrics, and so we use the execd plugin to shell out to a golang binary to process these metrics.

We found that our execd plugin was a bottleneck, but it only does very modest string manipulation. We think that the bottleneck comes mostly from deserializing and reserializing the metrics from STDIN to STDOUT.

We managed to get more throughput by creating multiple instances of our plugin in our `telegraf.conf` file. We create instances 1...N with names like `plugin-1` and `plugin-2`. We use the starlark processor to add a `pipeline` tag to each metric in round-robin fashion, like this:

```starlark
state = {"round_robin_count": 0}
def apply(metric):
num = (1 + state["round_robin_count"])
state["round_robin_count"] = num % pipeline_count
metric.tags["pipeline"] = "pipeline-{}".format(num)
return metric
```

So the pipeline is something like
Prometheus Input -> Starlark -> Execd Plugin {1...N} -> Output {1...N}

The pattern works well for us, but is noisy - the telegraf config has to be generated from a template and the `telegraf.conf` file has N mostly identical variants of the processor plugin and output plugin configurations.

Maybe telegraf itself would be able to do this with less complexity, since it could easily initialize multiple versions of a plugin from the same configuration and could probably route metrics between multiple instances without needing to create an ephemeral tag just for routing. For users who deal with a large metrics volume and encounter bottlenecks in their plugin pipeline, this could help them get greater throughput from their telegraf deployment without adding substantial complexity to their configuration.

### Expected behavior

We wanted to be able to parallelize some of our processor plugins and output plugins easily

### Actual behavior

We needed to do a fair bit of work to come up with a way to round-robin between multiple instances of the same plugin, and the resulting configuration file is large and contains a lot of duplication.

### Additional info

Thanks for reading!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.