Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka
behavior.on.version.conflict=warn still throws errors
- Vorherrschende Sprache
- Java
- Sterne
- 95
- Forks
- 61
- Ø Merge
- 11 Std. 52 Min.
- Gemergte PRs (30 T.)
- 7
Beschreibung
Hello 👋, I recently updated our connector to 4.1.0 and started getting these 👇 errors when I run a backfill. The version conflicts should be ignored and just log so the config is set to `"behavior.on.version.conflict": "warn",` but the connector still gets exception instead of warn log. Also one of the JMX metric we use to indicate if the connector is in fact running is `jmx.kafka.connect.total_record_failures` which also shows `34.8k`.
Am I missing some something or is this a bug? I saw https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/pull/413 and https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/pull/419 were fixed in V4.1.0 so this should have worked.
ERROR:
```
Jun 26 13:23:40.360 prod-kc-os-sink-track info java.lang.Exception: Encountered a version conflict when executing batch 601 of 49 records. Reporting this error to the errant record reporter and will keep an existing record. Rest status: 409, Action id: 48, Error message: [9103882]: version conflict, current version [2660808] is higher or equal to the one provided [2660808]
Jun 26 13:23:40.360 prod-kc-os-sink-track info [2026-06-26 07:53:40,353] ERROR Error encountered in task prod-kc-os-sink-track-0. Executing stage 'TASK_PUT' with class 'org.apache.kafka.connect.sink.SinkTask', where consumed record is {topic='event.backup.prod.etl.tracks', partition=0, offset=2660808, timestamp=1782459555748, timestampType=CreateTime}. (org.apache.kafka.connect.runtime.errors.LogReporter)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at java.base/java.lang.Thread.run(Unknown Source)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.call(BulkProcessor.java:342)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.call(BulkProcessor.java:361)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.execute(BulkProcessor.java:389)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.RetryUtil.callWithRetry(RetryUtil.java:97)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.lambda$execute$1(BulkProcessor.java:402)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.handleVersionConflict(BulkProcessor.java:439)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.sendToErrantRecordReporter(BulkProcessor.java:373)
```
Beitragsleitfaden
Rechercherichtung
Start by tracing the version-conflict path in BulkProcessor.java, especially BulkTask.call, execute, handleVersionConflict, and sendToErrantRecordReporter at the lines shown in the stack trace. Check how behavior.on.version.conflict=warn is read and applied, then reproduce the backfill case and verify that the conflict is logged without throwing an exception or producing the reported task failure.
Vom Indexierungsmodell aus dem Issue-Text verfasst.
Bewertung
- Tech-Stack
- java
- Bereich
- backend, distributed-systems
- Issue-Typ
- Bug
- Schwierigkeit
- 3/5
- Geschätzter Aufwand
- 1-2 Tage
- Aktivitätsstatus
- Ruhig
- Klarheit
- Größtenteils klar
- Anfängerfreundlichkeit
- 57/100