apache / apache/hudi

Add tool to capture earliest or latest offsets in kafka topics

Open
#14,670 0 comments 0 reactions 0 assignees View on GitHub
area:ingest from-jira priority:high type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

For bootstrapping cases using spark.write(), we need to capture offsets from kafka topic and use it as checkpoint for subsequent read from Kafka topics.

 

[https://github.com/apache/hudi/issues/1985]

We need to build this integration for smooth transition to deltastreamer.

 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1280
- Type: New Feature

---

## Comments

06/Dec/21 14:31;shivnarayan;CC [~rmahindra] ;;;

---

09/Dec/21 01:21;shivnarayan;[~vbalaji] : We already have a config to auto resetting offsets in kafka. Or are you referring to something else. can you throw some light please. ;;;

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing the related issue at https://github.com/apache/hudi/issues/1985 and the existing Kafka offset auto-reset configuration mentioned in the comments. Clarify how spark.write() bootstrapping should capture earliest or latest Kafka offsets and how those offsets become checkpoints for subsequent reads; the integration is complete when the transition to deltastreamer is supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka, spark
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.