Aiven-Open / Aiven-Open/http-connector-for-apache-kafka

Http Sink Connector retry messages when the service has a long-time response

未关闭
#265 2 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Java
星标
119
派生
63
PR 合并指标
30 天内没有已合并 PR

描述

I have to created sink connector with configuration like this
{
"connector.class": "io.aiven.kafka.connect.http.HttpSinkConnector",
"http.authorization.type": "none",
"tasks.max": "3",
"name": "{{connector-name}}",
"http.url": "{{service-endpoint-url}}",
"auto.commit.interval.ms": "15000",
"heartbeat.interval.ms": "15000",
"value.converter": "org.apache.kafka.connect.storage.StringConverter",
"retry.backoff.ms": "30000",
"http.ssl.trust.all.certs": "true",
"topics.regex": "{{topic-name}}",
"max.poll.interval.ms": "3600000"
}

I reviewed the service response log and observed that the duration ranged between 5 and 10 seconds, then I counted the number of requests and I found the number of requests was more than the number of messages. Assume that if I have 500 messages in Kafka, the number of requests is more than 500 requests and I find some messages will be sent duplicate.

Which config will fix this issue ?


Thank you

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。