[SUPPORT] Spark Write into MoR type hudi table small parquets issue + Athena Internal Error
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
**Describe the problem you faced**
Background:
Currently, I have around 100 mb data each day (batch process), so I am using the delete operation with broadcast join in spark to delete the unused data(data will be updated the next day), I try to avoid global index scanning, so I insert delta data into a new partition.
**Parquet size Issue:**
Set hoodie. shuffle. parallelism =2, does not change the **parquet size** when using **insert**.
It generates 40 parquets for one partitiom(daily), total size: 180 mb
**Athena Issue:**
Due to the small size of parquets, I implemented clustering (inline) with max commits =1 for test.
Athena Raises Error:
Generic_INTERNAL_ERROR: Can not read value at 0 in block -1 in S3/XXXXXX/XXX/XXXX/XXXXX/date=20XX-XX-XX.parquet.
**To Reproduce**
Steps to reproduce the behavior:
1. Hudi Options like below:
Table ddl (_rt, _ro) was created automatically when enabling the hive sync tool.

record keys are strings, partition keys are just dates in yyyy-mm-dd
2.

**Expected behavior**
I wish to keep my parquet size around 120 MB for each file (one partition) or two parquet files (around 64 MB).
**Environment Description**
* Hudi version : 0.13.0
* Spark version : 3.4.1
* Hive version : 0.13.1
* Hadoop version :
* Storage (HDFS/S3/GCS..) : S3
* Running on Docker? (yes/no) : NO
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the reported insert and inline-clustering workflow on Hudi 0.13.0 with Spark 3.4.1 and S3, using the configuration shown in the issue and date-partitioned data. Compare the generated Parquet sizes and test the clustered output through Athena. Done means isolating whether the small files or Athena read error is caused by configuration or a Hudi defect, with a documented resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100