influxdata / influxdata/telegraf

New plugin type: discovery plugins

Open
#13,569 5 comments 0 reactions 0 assignees View on GitHub
feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

When it is required to have certain Input plugin for each X of something currently it has to be done in the configuration management tool. This can be too limiting when running in a more dynamic environment such as clouds or kubernetes.

For instance there is a `postgres` Input plugin which requires database connection string. On the AWS cloud RDS databases can come and go and if task is to monitor all of them it is not straightforward how to achieve it.

### Expected behavior

Config option to instantiate plugins dynamically based on the output of some discovery process.

### Actual behavior

-

### Additional info

I didn't give much thought how exactly to achieve it, but as a rough idea:

Every discovery plugin returns equivalent of JSON array of maps. Each map contains configuration values to be used by plugins

```
[[discovery.exec]]
alias = "example"
# simple discovery plugin which just prints discovery result.
command = ["/bin/sh", "-c", "echo '[{host: \"xyz\", interface: \"eth0\"}, {host: \"abc\", interface: \"eth1\"}]'"]
```

Then plugins can be wired to the discovery plugin. When wired that can use map values from discovery:

```
[[input.ping]]
discovery = discovery.exec.example # identifier of the discovery plugin used to instantiate this plugin
alias = "${discovery.host}"
urls = ["${discovery.host}"]
interface = "${discovery.interface}"
```

Should "spawn" 2 `ping` plugins dynamically.

Discovery plugin runs on the the interval as any other plugins and plugins it spawns ideally should stay up unless discovery plugin changes output.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Telegraf's existing plugin lifecycle and the proposed discovery.exec and input.ping configuration examples. Define how discovery results instantiate and update plugins, including interval handling and retaining unchanged plugins. Done means a concrete design and implementation plan exists for dynamic plugin creation from discovery output.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.