influxdata / influxdata/telegraf

inputs.logstash: additional metrics

Open
#19,153 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

Monitoring logstash internal metrics

### Expected behavior

Telegraf should collect and expose all useful metrics from Logstash.

### Actual behavior

Several groups of Logstash metrics are not collected. This is likely a case of new metrics having been introduced in the API over time, and the plugin not updated to suit.

From [_node/stats/pipelines](https://www.elastic.co/docs/api/doc/logstash/operation/operation-nodestatspipelines) we collect
* `pipelines/*/events/*` (in/out/durations)
* `pipelines/*/queue/*`
* `pipelines/*/plugins/.../events/*` (in/out/success/failures per plugin)

Useful additions would include
* `pipelines/*/pipeline/*` (for workers, batch_size)
* `pipelines/*/flow/**` (for worker util%, concurrency, throughput)
* `pipelines/*/plugins/*/flow/**` (same per plugin)

The `workers` value is particularly useful for scaling other metrics like `duration_in_millis`, which convey poor information unless you know how many workers are active.

In passing, `pipelines/*/reloads` is collected but not exposed.

### Additional info

These metrics can be collected independently using an http input, but it's unweildy, and duplicates effort and throughput. We're using this simple config at the moment to partly fill the feature gap:

```
[[inputs.http]]
urls = ["http://localhost:9600/_node/stats/pipelines"]
method = "GET"
timeout = "5s"
data_format = "json"
name_override = "logstash_pipeline"
```

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by inspecting the inputs.logstash plugin and how it parses the _node/stats/pipelines response. Trace the existing pipeline, queue, and plugin event metrics, then determine how to expose workers, batch_size, flow metrics, per-plugin flow metrics, and reloads. Done means the requested Logstash metrics are collected and exposed without requiring a separate HTTP input.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.