apache / apache/hudi

[SUPPORT]

Open
#13,513 4 comments 0 reactions 0 assignees View on GitHub
area:cdc type:community-support
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I have an issue when reading CDC from a Hudi MOR (Merge-On-Read) table.
Even though I’ve configured the following options:

```
'hoodie.table.cdc.enabled': 'true',
'hoodie.table.cdc.supplemental.logging.mode': 'DATA_BEFORE_AFTER'
```
When I try to read CDC from that table, I encounter this error:

```
25/07/02 16:45:59 ERROR Executor: Exception in task 4.0 in stage 0.0 (TID 4)
java.io.FileNotFoundException: No such file or directory:
s3a://ndc-hudi/test3/partition_id=9/.8d65e8e2-551a-40d9-b95a-09e898dd84b4-0_20250702035214716.log.1_0-1035-8955.cdc
```
The issue is that Hudi has cleaned up all the CDC files that were too old.

Is there any Hudi configuration that allows me to continue compacting the MOR table periodically while still preserving all CDC files, so I can view changes at any point in time?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the two Hudi CDC configuration keys and the MOR CDC read and cleanup behavior described in the report. Check whether compaction and cleanup provide a supported retention setting for preserving CDC files; done means identifying the applicable configuration or clarifying that the requested retention is unsupported.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.