[SUPPORT] Hudi COW not honouring file size config
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
We are running a glue job to write COW hudi tables . It runs for every 4 hour. We have set file sizing config to 64 MB. But, recently we are seeing files upto 150 MB.
**To Reproduce**
hudi_options = {
"hoodie.datasource.write.partitionpath.urlencode": "true",
'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
'hoodie.datasource.write.reconcile.schema': 'true',
'hoodie.schema.on.read.enable': 'true',
'hoodie.table.base.file.format': 'PARQUET',
'hoodie.parquet.compression.codec': 'snappy',
"hoodie.datasource.write.hive_style_partitioning": "true",
"hoodie.datasource.hive_sync.enable": "true",
"hoodie.datasource.hive_sync.partition_extractor_class": "org.apache.hudi.hive.MultiPartKeysValueExtractor",
"hoodie.datasource.hive_sync.use_jdbc": "false",
"hoodie.datasource.hive_sync.mode": "hms",
"hoodie.datasource.hive_sync.support_timestamp": "true",
'hoodie.parquet.max.file.size': 67108864,
'hoodie.parquet.small.file.limit': 62914560,
'hoodie.insert.shuffle.parallelism': 500,
'hoodie.cleaner.commits.retained': 2
}
**Expected behavior**
The file sizes should not be greater than 64 MB
**Environment Description**
Hudi version :0.12.1
Spark version : 3.3
Hive version :
Hadoop version :
Storage (HDFS/S3/GCS..) : s3
Running on Docker? (yes/no) : no
* Storage (HDFS/S3/GCS..) :
* Running on Docker? (yes/no) :
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the Glue job described with Hudi 0.12.1 and Spark 3.3, using the listed COPY_ON_WRITE and parquet file-size settings. Compare the resulting S3 file sizes with the 64 MB expectation and inspect the file-sizing behavior around those configuration values; done means identifying why files exceed the configured maximum and verifying the expected limit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, spark
- Domain
- cloud, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100