elastic / elastic/logstash

Elasticsearch Output does not allow setting `retry_on_conflict` to `0` with data stream set to `true`

Open
#16,716 0 comments 0 reactions 0 assignees View on GitHub
bug status:needs-triage
Dominant language
Java
Stars
14.9k
Forks
3.5k
Avg merge
1d 4h
Merged PRs (30d)
88

Description

**Logstash information**:

Please include the following information:

1. Logstash version: 8.15.4
2. Logstash installation source: Docker
3. How is Logstash being run: Kubernetes

**Plugins installed**: (`bin/logstash-plugin list --verbose`)

**JVM** (e.g. `java -version`):

If the affected version of Logstash is 7.9 (or earlier), or if it is NOT using the bundled JDK or using the 'no-jdk' version in 7.10 (or higher), please provide the following information:

1. JVM version (`java -version`)
2. JVM installation source (e.g. from the Operating System's package manager, from source, etc).
3. Value of the `LS_JAVA_HOME` environment variable if set.

**OS version** (`uname -a` if on a Unix-like system):

**Description of the problem including expected versus actual behavior**:

When setting an Elasticsearch output with `data_stream => true`, you cannot set `retry_on_conflict => 0`.

This causes issues where integrations intentionally set `_id` fields and causes Elasticsearch to retry even when it is known that it will fail.

**Steps to reproduce**:

Please include a *minimal* but *complete* recreation of the problem,
including (e.g.) pipeline definition(s), settings, locale, etc. The easier
you make for us to reproduce it, the more likely that somebody will take the
time to look at it.

1. Create Elasticsearch output
2. Set `data_stream => true`
3. Set `retry_on_conflict => 0`
4. Observe log message stating that `retry_on_conflict => 0` is invalid for data streams.

**Provide logs (if relevant)**:

Contributor guide

Open the contributing guide

Research direction

Start with the Elasticsearch output configuration and reproduce the issue using data_stream = true and retry_on_conflict = 0. Trace the validation that rejects zero and verify completion when zero is accepted without the invalid-configuration message.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.