Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka

behavior.on.version.conflict=warn still throws errors

Offen
#428 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Java
Sterne
95
Forks
61
Ø Merge
11 Std. 52 Min.
Gemergte PRs (30 T.)
7

Beschreibung

Hello 👋, I recently updated our connector to 4.1.0 and started getting these 👇 errors when I run a backfill. The version conflicts should be ignored and just log so the config is set to `"behavior.on.version.conflict": "warn",` but the connector still gets exception instead of warn log. Also one of the JMX metric we use to indicate if the connector is in fact running is `jmx.kafka.connect.total_record_failures` which also shows `34.8k`.
Am I missing some something or is this a bug? I saw https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/pull/413 and https://github.com/Aiven-Open/opensearch-connector-for-apache-kafka/pull/419 were fixed in V4.1.0 so this should have worked.

ERROR:
```
Jun 26 13:23:40.360 prod-kc-os-sink-track info java.lang.Exception: Encountered a version conflict when executing batch 601 of 49 records. Reporting this error to the errant record reporter and will keep an existing record. Rest status: 409, Action id: 48, Error message: [9103882]: version conflict, current version [2660808] is higher or equal to the one provided [2660808]
Jun 26 13:23:40.360 prod-kc-os-sink-track info [2026-06-26 07:53:40,353] ERROR Error encountered in task prod-kc-os-sink-track-0. Executing stage 'TASK_PUT' with class 'org.apache.kafka.connect.sink.SinkTask', where consumed record is {topic='event.backup.prod.etl.tracks', partition=0, offset=2660808, timestamp=1782459555748, timestampType=CreateTime}. (org.apache.kafka.connect.runtime.errors.LogReporter)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at java.base/java.lang.Thread.run(Unknown Source)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.call(BulkProcessor.java:342)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.call(BulkProcessor.java:361)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.execute(BulkProcessor.java:389)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.RetryUtil.callWithRetry(RetryUtil.java:97)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.lambda$execute$1(BulkProcessor.java:402)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.handleVersionConflict(BulkProcessor.java:439)
Jun 26 13:23:40.360 prod-kc-os-sink-track info at io.aiven.kafka.connect.opensearch.bulk.BulkProcessor$BulkTask.sendToErrantRecordReporter(BulkProcessor.java:373)
```

Beitragsleitfaden

Beitragsleitfaden öffnen

Rechercherichtung

Start by tracing the version-conflict path in BulkProcessor.java, especially BulkTask.call, execute, handleVersionConflict, and sendToErrantRecordReporter at the lines shown in the stack trace. Check how behavior.on.version.conflict=warn is read and applied, then reproduce the backfill case and verify that the conflict is logged without throwing an exception or producing the reported task failure.

Vom Indexierungsmodell aus dem Issue-Text verfasst.

Bewertung

Tech-Stack
java
Bereich
backend, distributed-systems
Issue-Typ
Bug
Schwierigkeit
3/5
Geschätzter Aufwand
1-2 Tage
Aktivitätsstatus
Ruhig
Klarheit
Größtenteils klar
Anfängerfreundlichkeit
57/100

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.