[SUPPORT] Process Fallout: Unable to upsert due to continues Failure in Compaction
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**Describe the problem you faced**
Unable to process data into the Hudi tables as inline Compaction is failing.
**To Reproduce**
Previous **hoodie.compact.inline.max.delta.commits** was 6. We changed it to 10.
Hudi MOR Table configuration:
`
{
'hoodie.table.name': asset,
'hoodie.datasource.write.recordkey.field': id,
'hoodie.datasource.write.table.name': asset,
'hoodie.upsert.shuffle.parallelism': 400,
'hoodie.keep.max.commits': 1000,
'hoodie.keep.min.commits': 990,
'hoodie.compact.inline.max.delta.commits': 10,
'hoodie.clean.automatic': 'false',
'hoodie.clean.async': 'false'
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.write.precombine.field': 'cdc_timestamp',
'hoodie.datasource.write.table.type': 'MERGE_ON_READ',
'hoodie.cleaner.policy': 'KEEP_LATEST_COMMITS',
'hoodie.schema.on.read.enable' : "true",
'hoodie.datasource.write.reconcile.schema' : "true",
'hoodie.compact.inline' : "true",
'hoodie.datasource.write.keygenerator.class': 'org.apache.hudi.keygen.NonpartitionedKeyGenerator'
}
`
Error:
`
org.apache.hudi.exception.HoodieCompactionException: Could not compact s3a://*************************/hudi_tables/assets/asset
`
**Environment Description**
* Hudi version : 0.13.0
* Spark version : 3.3.1
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
[HudiCompactionIssue20231003.txt](https://github.com/apache/hudi/files/12812602/HudiCompactionIssue20231003.txt)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading the attached HudiCompactionIssue20231003.txt stack trace and reviewing the listed Hudi 0.13.0, Spark 3.3.1, S3, and inline-compaction settings. No source file or test is named, so the first step is to identify the failing compaction path and establish a reproducible case; done requires determining why the upsert compaction fails and documenting or testing the fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100