Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka

Connector periodically issues warnings: "Commit of offsets timed out"

Abierto
#272 2 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
95
Forks
61
Merge medio
11 h 52 min
PR fusionados (30 d)
7

Descripción

We are in the process of evaluating and establishing a CDC system streaming change events from our databases into an Opensearch deployment.

The Opensearch connector eventually pushing aggregated topics into Opensearch periodically issues warnings like this:

```
...

2024-03-16 22:38:59,742 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

2024-03-16 22:40:01,936 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

2024-03-16 22:41:03,366 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

...
```

These messages occur roughly every minute.

Our connector config basically looks like this:

```
spec:
class: io.aiven.kafka.connect.opensearch.OpensearchSinkConnector
config:
batch.size: 1000
behavior.on.malformed.documents: warn
behavior.on.null.values: delete
behavior.on.version.conflict: warn
connection.password: ${secrets:debezium/opensearch-credentials:password}
connection.url: https://debezium-opensearch-nodes.debezium.svc:9200
connection.username: ${secrets:debezium/opensearch-credentials:username}
errors.deadletterqueue.context.headers.enable: true
errors.deadletterqueue.topic.name: myjack.index.dl
errors.deadletterqueue.topic.replication.factor: 3
errors.tolerance: all
flush.timeout.ms: 30000
index.write.method: upsert
key.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: false
key.ignore: false
key.ignore.id.strategy: record.key
max.buffered.records: 20000
schema.ignore: "true"
topics:
transforms: extractKey
transforms.extractKey.field: id
transforms.extractKey.type: org.apache.kafka.connect.transforms.ExtractField$Key
type.name: _doc
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: false
```

Is this something noteworthy, does it indicate an issue? What causes these periodic messages?

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.