influxdata / influxdata/telegraf
inputs.logstash: additional metrics
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
Monitoring logstash internal metrics
### Expected behavior
Telegraf should collect and expose all useful metrics from Logstash.
### Actual behavior
Several groups of Logstash metrics are not collected. This is likely a case of new metrics having been introduced in the API over time, and the plugin not updated to suit.
From [_node/stats/pipelines](https://www.elastic.co/docs/api/doc/logstash/operation/operation-nodestatspipelines) we collect
* `pipelines/*/events/*` (in/out/durations)
* `pipelines/*/queue/*`
* `pipelines/*/plugins/.../events/*` (in/out/success/failures per plugin)
Useful additions would include
* `pipelines/*/pipeline/*` (for workers, batch_size)
* `pipelines/*/flow/**` (for worker util%, concurrency, throughput)
* `pipelines/*/plugins/*/flow/**` (same per plugin)
The `workers` value is particularly useful for scaling other metrics like `duration_in_millis`, which convey poor information unless you know how many workers are active.
In passing, `pipelines/*/reloads` is collected but not exposed.
### Additional info
These metrics can be collected independently using an http input, but it's unweildy, and duplicates effort and throughput. We're using this simple config at the moment to partly fill the feature gap:
```
[[inputs.http]]
urls = ["http://localhost:9600/_node/stats/pipelines"]
method = "GET"
timeout = "5s"
data_format = "json"
name_override = "logstash_pipeline"
```
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the inputs.logstash plugin and how it parses the _node/stats/pipelines response. Trace the existing pipeline, queue, and plugin event metrics, then determine how to expose workers, batch_size, flow metrics, per-plugin flow metrics, and reloads. Done means the requested Logstash metrics are collected and exposed without requiring a separate HTTP input.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100