apache / apache/hudi

Initial Kafka Global Offsets in Hudi Kafka Sink Connector

Open
#15,328 0 comments 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
engine:kafka-connect from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Hi team,
I am trying to run Hudi Sink Connector with Kafka Connect. When the connectors starts, it starts the Transaction coordinator which initialises the global committed  offsets from the Hudi commit file. When its a first time run, there is no commit file and hence it outputs
[2022-08-08 19:58:20,529] INFO Hoodie Extra Metadata from latest commit is absent (org.apache.hudi.connect.writers.KafkaConnectTransactionServices:147)
But say in first time, the earliest kafka offset is not 0, then the process keeps on running the commit timelines. Ideally, the global offsets, at first run, should be set to the earliest kafka offset.
As per the current implementation, the participant checks the local offset with coordinator offset and when its a mismatch, it sets to 0. But this breaks, when its a fresh run and the global kafka commited offset is not 0

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-4575
- Type: Bug
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.