elastic / elastic/logstash

Centralised pipelines don't honour the same configuration directives as static ones

Open
#16,729 0 comments 0 reactions 0 assignees View on GitHub
bug status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

**Logstash information**:

Please include the following information:

1. Logstash version: **8.15.3**
2. Logstash installation source: **docker**
3. How is Logstash being run: **docker**

**OS version** (`uname -a` if on a Unix-like system):
ubuntu 22.04

**Description of the problem including expected versus actual behaviour**:
Raised off https://discuss.elastic.co/t/custom-pipeline-settings-with-centralised-pipeline-management/371001

When using the API to create centralised logstash pipelines, you can add all the [settings documented](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html), however the majority of these settings are not honoured when the pipeline runs:
```
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.enable'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.retain.age'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.flush_interval'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.storage_policy'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'queue.drain'
```

This means that centralised pipelines can't be used to replace all but the most basic of logstash pipelines.

It's unclear why the settings wouldn't be honoured, but if not they should be rejected by the API, and the documentation should reflect this limitation.

For completeness the discuss article above also mentions:
* If you subsequently edit them through Kibana, all those custom settings are lost.
* The "Last Modified" time can't be set (and doesn't default to) "now" - you have to enter the entire timestamp each time.

**Steps to reproduce**:

Create a centralised pipeline with custom settings. Not that your setting don't apply.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the issue through the centralised pipeline API with the documented settings, then trace the Elasticsearch source handling that logs unsupported pipeline settings. Compare the API's accepted settings with those applied at runtime and check the Kibana editing behavior. Done means supported settings work consistently, unsupported settings are rejected, or the limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.