Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka
Connector periodically issues warnings: "Commit of offsets timed out"
- Lenguaje dominante
- Java
- Estrellas
- 95
- Forks
- 61
- Merge medio
- 11 h 52 min
- PR fusionados (30 d)
- 7
Descripción
We are in the process of evaluating and establishing a CDC system streaming change events from our databases into an Opensearch deployment.
The Opensearch connector eventually pushing aggregated topics into Opensearch periodically issues warnings like this:
```
...
2024-03-16 22:38:59,742 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]
2024-03-16 22:40:01,936 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]
2024-03-16 22:41:03,366 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]
...
```
These messages occur roughly every minute.
Our connector config basically looks like this:
```
spec:
class: io.aiven.kafka.connect.opensearch.OpensearchSinkConnector
config:
batch.size: 1000
behavior.on.malformed.documents: warn
behavior.on.null.values: delete
behavior.on.version.conflict: warn
connection.password: ${secrets:debezium/opensearch-credentials:password}
connection.url: https://debezium-opensearch-nodes.debezium.svc:9200
connection.username: ${secrets:debezium/opensearch-credentials:username}
errors.deadletterqueue.context.headers.enable: true
errors.deadletterqueue.topic.name: myjack.index.dl
errors.deadletterqueue.topic.replication.factor: 3
errors.tolerance: all
flush.timeout.ms: 30000
index.write.method: upsert
key.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: false
key.ignore: false
key.ignore.id.strategy: record.key
max.buffered.records: 20000
schema.ignore: "true"
topics:
transforms: extractKey
transforms.extractKey.field: id
transforms.extractKey.type: org.apache.kafka.connect.transforms.ExtractField$Key
type.name: _doc
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: false
```
Is this something noteworthy, does it indicate an issue? What causes these periodic messages?
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.