elastic / elastic/logstash

Adapt pipeline specific APIs to support multi pipeline

Open
#6,531 7 comments 0 reactions 0 assignees View on GitHub
code cleanup discuss enhancement multiple pipelines v5.5.0
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
19h 14m
Merged PRs (30d)
63

Description

Currently, some of the apis assume only 1 pipeline exists. With the upcoming multiple pipeline feature, this needs to be addressed, with the caveat of keeping backwards compatibility.

Single Pipeline Assumptions:

## 1. Pipeline Info API

`GET /_node/pipeline`

```json
{
"host": "Joaos-MBP-5.lan",
"version": "6.0.0-alpha1",
"http_address": "127.0.0.1:9600",
"id": "3d3147fe-eb86-45e9-9d13-2fb83a7c1550",
"name": "Joaos-MBP-5.lan",
"pipeline": {
"workers": 4,
"batch_size": 125,
"batch_delay": 5,
"config_reload_automatic": false,
"config_reload_interval": 3
}
}
```

## 2. Pipeline Stats API

`GET /_node/stats/pipeline`

```json
{
"pipeline": {
"events": {
"duration_in_millis": 7863504,
"in": 100,
"filtered": 100,
"out": 100
},
"plugins": {
"inputs": [],
"filters": [
{
"id": "grok_20e5cb7f7c9e712ef9750edf94aefb465e3e361b-2",
"events": {
"duration_in_millis": 48,
"in": 100,
"out": 100
},
"matches": 100,
"patterns_per_field": {
"message": 1
},
"name": "grok"
},
{
"id": "geoip_20e5cb7f7c9e712ef9750edf94aefb465e3e361b-3",
"events": {
"duration_in_millis": 141,
"in": 100,
"out": 100
},
"name": "geoip"
}
],
"outputs": [
{
"id": "20e5cb7f7c9e712ef9750edf94aefb465e3e361b-4",
"events": {
"in": 100,
"out": 100
},
"name": "elasticsearch"
}
]
},
"reloads": {
"last_error": null,
"successes": 0,
"last_success_timestamp": null,
"last_failure_timestamp": null,
"failures": 0
}
}
```

## 3. Node Stats API

The node stats includes data from the pipeline stats api:

`GET /_node/stats/pipeline`

```json
{
"host": "Joaos-MBP-5.lan",
"version": "6.0.0-alpha1",
"http_address": "127.0.0.1:9600",
"id": "3d3147fe-eb86-45e9-9d13-2fb83a7c1550",
"name": "Joaos-MBP-5.lan",
"jvm": {
# ...
"uptime_in_millis": 108897
},
"process": {
#...
},
"pipeline": {
"events": {
"duration_in_millis": 0,
"in": 0,
"filtered": 0,
"out": 0
},
"plugins": {
"inputs": [],
"filters": [],
"outputs": [
{
"id": "92ddb0615336293c1757cac81c3bebfa19985e68-2",
"name": "stdout"
}
]
},
"reloads": {
"last_error": null,
"successes": 0,
"last_success_timestamp": null,
"last_failure_timestamp": null,
"failures": 0
},
"queue": {
"type": "memory"
}
},
"reloads": {
"successes": 0,
"failures": 0
},
"os": {}
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.