Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka

Connector periodically issues warnings: "Commit of offsets timed out"

Aperta
#272 2 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Java
Stelle
95
Fork
61
Merge medio
11h 52m
PR unite (30g)
7

Descrizione

We are in the process of evaluating and establishing a CDC system streaming change events from our databases into an Opensearch deployment.

The Opensearch connector eventually pushing aggregated topics into Opensearch periodically issues warnings like this:

```
...

2024-03-16 22:38:59,742 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

2024-03-16 22:40:01,936 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

2024-03-16 22:41:03,366 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

...
```

These messages occur roughly every minute.

Our connector config basically looks like this:

```
spec:
class: io.aiven.kafka.connect.opensearch.OpensearchSinkConnector
config:
batch.size: 1000
behavior.on.malformed.documents: warn
behavior.on.null.values: delete
behavior.on.version.conflict: warn
connection.password: ${secrets:debezium/opensearch-credentials:password}
connection.url: https://debezium-opensearch-nodes.debezium.svc:9200
connection.username: ${secrets:debezium/opensearch-credentials:username}
errors.deadletterqueue.context.headers.enable: true
errors.deadletterqueue.topic.name: myjack.index.dl
errors.deadletterqueue.topic.replication.factor: 3
errors.tolerance: all
flush.timeout.ms: 30000
index.write.method: upsert
key.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: false
key.ignore: false
key.ignore.id.strategy: record.key
max.buffered.records: 20000
schema.ignore: "true"
topics:
transforms: extractKey
transforms.extractKey.field: id
transforms.extractKey.type: org.apache.kafka.connect.transforms.ExtractField$Key
type.name: _doc
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: false
```

Is this something noteworthy, does it indicate an issue? What causes these periodic messages?

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.