Elasticsearch Output does not allow setting `retry_on_conflict` to `0` with data stream set to `true`
- Dominant language
- Java
- Stars
- 14.9k
- Forks
- 3.5k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 88
Description
**Logstash information**:
Please include the following information:
1. Logstash version: 8.15.4
2. Logstash installation source: Docker
3. How is Logstash being run: Kubernetes
**Plugins installed**: (`bin/logstash-plugin list --verbose`)
**JVM** (e.g. `java -version`):
If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:
1. JVM version (`java -version`)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the `LS_JAVA_HOME` environment variable if set.
**OS version** (`uname -a` if on a Unix-like system):
**Description of the problem including expected versus actual behavior**:
When setting an Elasticsearch output with `data_stream => true`, you cannot set `retry_on_conflict => 0`.
This causes issues where integrations intentionally set `_id` fields and causes Elasticsearch to retry even when it is known that it will fail.
**Steps to reproduce**:
Please include a *minimal* but *complete* recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.
1. Create Elasticsearch output
2. Set `data_stream => true`
3. Set `retry_on_conflict => 0`
4. Observe log message stating that `retry_on_conflict => 0` is invalid for data streams.
**Provide logs (if relevant)**:
Contributor guide
Research direction
Start with the Elasticsearch output configuration and reproduce the issue using data_stream = true and retry_on_conflict = 0. Trace the validation that rejects zero and verify completion when zero is accepted without the invalid-configuration message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100