apache / apache/paimon

[Feature] Schema Change can be generated by data in CDC ingestion

Open
#1,313 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

Especially in Kafka, there may be disorder, and the Schema Change may arrive later than the data. If there is a checkpoint difference between them, it may cause data blockage.

For schema changes, they can actually be generated from data and do not necessarily rely on cdc.

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

No files, tests, or entry points are identified in the issue. Start by locating the CDC ingestion handling for schema changes and the behavior around out-of-order Kafka data; done should include tests showing that schema changes can be generated from data without blocking ingestion.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kafka
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.