Aiven-Open / Aiven-Open/opensearch-connector-for-apache-kafka

OS Connector not sending messages to Dead letter Queue topic

Abierto
#278 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Java
Estrellas
95
Forks
61
Merge medio
11 h 52 min
PR fusionados (30 d)
7

Descripción

Hello,

we have Opensearch connector configured to send data to OS. However the records that are failing the due to serialization error is simply getting dropped and lost instead of sending them to DLQ topic.
Can someone please help us figure out what could be the probable cause or even if avien support DLQ ?

Please find the connector config below

{
"connector.class": "io.aiven.kafka.connect.opensearch.OpensearchSinkConnector",
"type.name": "_doc",
"behavior.on.null.values": "ignore",
"auto.create.indices.at.start": "false",
"tasks.max": "150",
"connection.timeout.ms": "3000000",
"drop.invalid.message": "false",
"max.retries": "5",
"key.ignore": "true",
"retry.backoff.ms": "30000",
"errors.deadletterqueue.context.headers.enable": "true",
"errors.deadletterqueue.topic.replication.factor": "3",
"read.timeout.ms": "300000",
"topics": "b6vv_onevz_nsacxp_payload_log-grp1",
"batch.size": "300",
"max.in.flight.requests": "4",
"schema.ignore": "true",
"behavior.on.malformed.documents": "REPORT",
"flush.timeout.ms": "300000",
"max.buffered.events": "300",
"errors.deadletterqueue.topic.name": "dlq_b6vv_payload_log-grp1_test",
"name": "eks-flb-msk-os-b6vv-onevz-nsacxp-payload-log-grp1-2-conn-sink",
"errors.tolerance": "all",
"connection.url": "https://vpc-vz-b6vv-nextgen-li6-7omprnsqqujw7ndad5yufgox7a.us-east-1.es.amazonaws.com",
"linger.ms": "3000"
}

Thanks

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.