Centralised pipelines don't honour the same configuration directives as static ones
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
**Logstash information**:
Please include the following information:
1. Logstash version: **8.15.3**
2. Logstash installation source: **docker**
3. How is Logstash being run: **docker**
**OS version** (`uname -a` if on a Unix-like system):
ubuntu 22.04
**Description of the problem including expected versus actual behaviour**:
Raised off https://discuss.elastic.co/t/custom-pipeline-settings-with-centralised-pipeline-management/371001
When using the API to create centralised logstash pipelines, you can add all the [settings documented](https://www.elastic.co/guide/en/logstash/current/logstash-settings-file.html), however the majority of these settings are not honoured when the pipeline runs:
```
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.enable'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.retain.age'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.flush_interval'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'dead_letter_queue.storage_policy'
[logstash.configmanagement.elasticsearchsource] Ignoring unsupported or unknown pipeline settings 'queue.drain'
```
This means that centralised pipelines can't be used to replace all but the most basic of logstash pipelines.
It's unclear why the settings wouldn't be honoured, but if not they should be rejected by the API, and the documentation should reflect this limitation.
For completeness the discuss article above also mentions:
* If you subsequently edit them through Kibana, all those custom settings are lost.
* The "Last Modified" time can't be set (and doesn't default to) "now" - you have to enter the entire timestamp each time.
**Steps to reproduce**:
Create a centralised pipeline with custom settings. Not that your setting don't apply.
Contributor guide
Research direction
Start by reproducing the issue through the centralised pipeline API with the documented settings, then trace the Elasticsearch source handling that logs unsupported pipeline settings. Compare the API's accepted settings with those applied at runtime and check the Kibana editing behavior. Done means supported settings work consistently, unsupported settings are rejected, or the limitation is documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100