[SUPPORT]
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I have an issue when reading CDC from a Hudi MOR (Merge-On-Read) table.
Even though I’ve configured the following options:
```
'hoodie.table.cdc.enabled': 'true',
'hoodie.table.cdc.supplemental.logging.mode': 'DATA_BEFORE_AFTER'
```
When I try to read CDC from that table, I encounter this error:
```
25/07/02 16:45:59 ERROR Executor: Exception in task 4.0 in stage 0.0 (TID 4)
java.io.FileNotFoundException: No such file or directory:
s3a://ndc-hudi/test3/partition_id=9/.8d65e8e2-551a-40d9-b95a-09e898dd84b4-0_20250702035214716.log.1_0-1035-8955.cdc
```
The issue is that Hudi has cleaned up all the CDC files that were too old.
Is there any Hudi configuration that allows me to continue compacting the MOR table periodically while still preserving all CDC files, so I can view changes at any point in time?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the two Hudi CDC configuration keys and the MOR CDC read and cleanup behavior described in the report. Check whether compaction and cleanup provide a supported retention setting for preserving CDC files; done means identifying the applicable configuration or clarifying that the requested retention is unsupported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, stream-processing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100