elastic / elastic/integrations
[Logstash Integration]: Collect node-level aggregate flow metrics with the Elastic Agent Logstash integration
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 225
Description
### Integration Name
Logstash [packages/logstash]
### Dataset Name
logstash.node
### Integration Version
2.11.1
### Agent Version
9.4.4
### OS Version and Architecture
Linux x86_64
### User Goal
Monitor overall Logstash node throughput and backpressure from a single view in Kibana without aggregating across 60+ individual pipeline documents. The `/_node/stats` API returns node-level aggregate flow metrics (`input_throughput`, `filter_throughput`, `output_throughput`, `queue_backpressure`, `worker_concurrency`) that give a single-pane view of node health.
### Existing Features
The integration collects per-pipeline flow metrics into `metrics-logstash.pipeline-*` but nothing in any dataset captures the top-level `flow` key from `/_node/stats`. The `metrics-logstash.node-*` data stream maps JVM, process, OS, events, and queue fields but has no `flow` property in its component template. Field caps query against `metrics-logstash.*/_field_caps?fields=logstash.node.stats.flow*` returns empty.
### What did you see?
Logstash API returns node-level flow with time-windowed values:
```
curl -s http://localhost:9600/_node/stats/flow | python3 -m json.tool
```
```
{
"flow": {
"input_throughput": { "current": 0.0, "last_1_minute": 0.0, "lifetime": 0.0 },
"filter_throughput": { "current": 0.0, "last_1_minute": 0.0, "lifetime": 0.0 },
"output_throughput": { "current": 0.0, "last_1_minute": 0.0, "lifetime": 0.0 },
"queue_backpressure":{ "current": 0.0, "last_1_minute": 0.0, "lifetime": 0.0 },
"worker_concurrency":{ "current": 0.0004, "last_1_minute": 0.0002, "lifetime": 0.0002 }
}
}
```
None of these fields are present in `metrics-logstash.node-*`.
### Anything else?
API reference: https://www.elastic.co/docs/api/doc/logstash/operation/operation-nodestatsflow
Related prior ER: https://github.com/elastic/enhancements/issues/23493
Contributor guide
Research direction
Inspect packages/logstash and the metrics-logstash.node data stream's component template; start by comparing its existing mappings with the /_node/stats/flow response. Add the node flow fields and verify they appear in field caps for metrics-logstash.node-*.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 70/100