apache / apache/hudi

Hudi 1.0.2 - Cleaner deletes log files scheduled for compaction

Open
#19,290 18 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
Cleaner deleted log files scheduled for compaction, resulting in compaction failure:

java.io.FileNotFoundException: No such file or directory: s3a://datalake/raw/hudi_table/hudi_partition/.00000003-0000-0000-0000-000000000000-0_20260703064612810.log.1_0-1-4

**What you expected:**
I see that cleaner has an explicit check that a log file shouldn't be in the compaction request file for it to be cleaned. This should've prevented it from deleting the file.

**Steps to reproduce:**
- Have a log file which has an instant time older than the base file
- Run compaction scheduling, it will pick up this log file
- Run cleaner, it will delete this log file
- Run compaction execution, it will fail

### Environment

**Hudi version:** 1.0.2
**Query engine:** (Spark/Flink/Trino etc): Flink writer, Spark table services
**Relevant configs:**

```
"--hoodie-conf", "hoodie.datasource.write.recordkey.field={{ params.table_record_key_fields }}",
"--hoodie-conf", "hoodie.bucket.index.hash.field={{ params.table_record_key_fields }}",
"--hoodie-conf", "hoodie.table.type=MERGE_ON_READ",
"--hoodie-conf", "hoodie.write.status.storage.level=MEMORY_AND_DISK_SER",
"--hoodie-conf", "hoodie.write.concurrency.mode=NON_BLOCKING_CONCURRENCY_CONTROL",
"--hoodie-conf", "hoodie.write.lock.provider=org.apache.hudi.aws.transaction.lock.DynamoDBBasedLockProvider",
"--hoodie-conf", "hoodie.write.lock.dynamodb.table={{ params.dynamodb_table }}",
"--hoodie-conf", "hoodie.write.lock.dynamodb.partition_key={{ params.table_name }}",
"--hoodie-conf", "hoodie.index.type=BUCKET",
"--hoodie-conf", "hoodie.metadata.enable=false",
"--hoodie-conf", "hoodie.index.bucket.engine=SIMPLE",
"--hoodie-conf", "hoodie.bucket.index.num.buckets=4",
"--hoodie-conf", "hoodie.clean.failed.writes.policy=LAZY",
"--hoodie-conf", "hoodie.clean.policy=KEEP_LATEST_BY_HOURS",
"--hoodie-conf", "hoodie.clean.hours.retained=24"

```

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the described sequence: schedule compaction for an old log file, run the cleaner, then execute compaction. Trace the cleaner's explicit compaction-request exclusion and the corresponding compaction scheduling and execution entry points. Done means the scheduled log file remains available and compaction completes without FileNotFoundException.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.