elastic / elastic/integrations

[Logstash Integration]: Collect node-level aggregate flow metrics with the Elastic Agent Logstash integration

Open
#21,163 0 comments 0 reactions 0 assignees View on GitHub
needs:triage
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
2d 17h
Merged PRs (30d)
225

Description

### Integration Name

Logstash [packages/logstash]

### Dataset Name

logstash.node

### Integration Version

2.11.1

### Agent Version

9.4.4

### OS Version and Architecture

Linux x86_64

### User Goal

Monitor overall Logstash node throughput and backpressure from a single view in Kibana without aggregating across 60+ individual pipeline documents. The `/_node/stats` API returns node-level aggregate flow metrics (`input_throughput`, `filter_throughput`, `output_throughput`, `queue_backpressure`, `worker_concurrency`) that give a single-pane view of node health.

### Existing Features

The integration collects per-pipeline flow metrics into `metrics-logstash.pipeline-*` but nothing in any dataset captures the top-level `flow` key from `/_node/stats`. The `metrics-logstash.node-*` data stream maps JVM, process, OS, events, and queue fields but has no `flow` property in its component template. Field caps query against `metrics-logstash.*/_field_caps?fields=logstash.node.stats.flow*` returns empty.

### What did you see?

Logstash API returns node-level flow with time-windowed values:

```
curl -s http://localhost:9600/_node/stats/flow | python3 -m json.tool
```

```
{
"flow": {
"input_throughput": { "current": 0.0, "last_1_minute": 0.0, "lifetime": 0.0 },
"filter_throughput": { "current": 0.0, "last_1_minute": 0.0, "lifetime": 0.0 },
"output_throughput": { "current": 0.0, "last_1_minute": 0.0, "lifetime": 0.0 },
"queue_backpressure":{ "current": 0.0, "last_1_minute": 0.0, "lifetime": 0.0 },
"worker_concurrency":{ "current": 0.0004, "last_1_minute": 0.0002, "lifetime": 0.0002 }
}
}
```

None of these fields are present in `metrics-logstash.node-*`.

### Anything else?

API reference: https://www.elastic.co/docs/api/doc/logstash/operation/operation-nodestatsflow

Related prior ER: https://github.com/elastic/enhancements/issues/23493

Contributor guide

Open the contributing guide

Research direction

Inspect packages/logstash and the metrics-logstash.node data stream's component template; start by comparing its existing mappings with the /_node/stats/flow response. Add the node flow fields and verify they appear in field caps for metrics-logstash.node-*.

Written by the indexing model from the issue text.

Assessment

Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.