Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka

Connector periodically issues warnings: "Commit of offsets timed out"

Ouverte
#272 2 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Java
Étoiles
95
Forks
61
Merge moyen
11 h 52 min
PR mergées (30 j)
7

Description

We are in the process of evaluating and establishing a CDC system streaming change events from our databases into an Opensearch deployment.

The Opensearch connector eventually pushing aggregated topics into Opensearch periodically issues warnings like this:

```
...

2024-03-16 22:38:59,742 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

2024-03-16 22:40:01,936 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

2024-03-16 22:41:03,366 WARN [connector-opensearch|task-0] WorkerSinkTask{id=connector-opensearch-0} Commit of offsets timed out (org.apache.kafka.connect.runtime.WorkerSinkTask) [task-thread-connector-opensearch-0]

...
```

These messages occur roughly every minute.

Our connector config basically looks like this:

```
spec:
class: io.aiven.kafka.connect.opensearch.OpensearchSinkConnector
config:
batch.size: 1000
behavior.on.malformed.documents: warn
behavior.on.null.values: delete
behavior.on.version.conflict: warn
connection.password: ${secrets:debezium/opensearch-credentials:password}
connection.url: https://debezium-opensearch-nodes.debezium.svc:9200
connection.username: ${secrets:debezium/opensearch-credentials:username}
errors.deadletterqueue.context.headers.enable: true
errors.deadletterqueue.topic.name: myjack.index.dl
errors.deadletterqueue.topic.replication.factor: 3
errors.tolerance: all
flush.timeout.ms: 30000
index.write.method: upsert
key.converter: org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable: false
key.ignore: false
key.ignore.id.strategy: record.key
max.buffered.records: 20000
schema.ignore: "true"
topics:
transforms: extractKey
transforms.extractKey.field: id
transforms.extractKey.type: org.apache.kafka.connect.transforms.ExtractField$Key
type.name: _doc
value.converter: org.apache.kafka.connect.json.JsonConverter
value.converter.schemas.enable: false
```

Is this something noteworthy, does it indicate an issue? What causes these periodic messages?

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.