Hudi 1.0.2 - Cleaner deletes log files scheduled for compaction
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
Cleaner deleted log files scheduled for compaction, resulting in compaction failure:
java.io.FileNotFoundException: No such file or directory: s3a://datalake/raw/hudi_table/hudi_partition/.00000003-0000-0000-0000-000000000000-0_20260703064612810.log.1_0-1-4
**What you expected:**
I see that cleaner has an explicit check that a log file shouldn't be in the compaction request file for it to be cleaned. This should've prevented it from deleting the file.
**Steps to reproduce:**
- Have a log file which has an instant time older than the base file
- Run compaction scheduling, it will pick up this log file
- Run cleaner, it will delete this log file
- Run compaction execution, it will fail
### Environment
**Hudi version:** 1.0.2
**Query engine:** (Spark/Flink/Trino etc): Flink writer, Spark table services
**Relevant configs:**
```
"--hoodie-conf", "hoodie.datasource.write.recordkey.field={{ params.table_record_key_fields }}",
"--hoodie-conf", "hoodie.bucket.index.hash.field={{ params.table_record_key_fields }}",
"--hoodie-conf", "hoodie.table.type=MERGE_ON_READ",
"--hoodie-conf", "hoodie.write.status.storage.level=MEMORY_AND_DISK_SER",
"--hoodie-conf", "hoodie.write.concurrency.mode=NON_BLOCKING_CONCURRENCY_CONTROL",
"--hoodie-conf", "hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider",
"--hoodie-conf", "hoodie.write.lock.dynamodb.table={{ params.dynamodb_table }}",
"--hoodie-conf", "hoodie.write.lock.dynamodb.partition_key={{ params.table_name }}",
"--hoodie-conf", "hoodie.index.type=BUCKET",
"--hoodie-conf", "hoodie.metadata.enable=false",
"--hoodie-conf", "hoodie.index.bucket.engine=SIMPLE",
"--hoodie-conf", "hoodie.bucket.index.num.buckets=4",
"--hoodie-conf", "hoodie.clean.failed.writes.policy=LAZY",
"--hoodie-conf", "hoodie.clean.policy=KEEP_LATEST_BY_HOURS",
"--hoodie-conf", "hoodie.clean.hours.retained=24"
```
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the described sequence: schedule compaction for an old log file, run the cleaner, then execute compaction. Trace the cleaner's explicit compaction-request exclusion and the corresponding compaction scheduling and execution entry points. Done means the scheduled log file remains available and compaction completes without FileNotFoundException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100