apache / apache/hudi

[SUPPORT] Spark Write into MoR type hudi table small parquets issue + Athena Internal Error

Open
#10,716 10 comments 0 reactions 0 assignees View on GitHub
area:storage priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

**Describe the problem you faced**

Background:
Currently, I have around 100 mb data each day (batch process), so I am using the delete operation with broadcast join in spark to delete the unused data(data will be updated the next day), I try to avoid global index scanning, so I insert delta data into a new partition.

**Parquet size Issue:**
Set hoodie. shuffle. parallelism =2, does not change the **parquet size** when using **insert**.
It generates 40 parquets for one partitiom(daily), total size: 180 mb

**Athena Issue:**
Due to the small size of parquets, I implemented clustering (inline) with max commits =1 for test.

Athena Raises Error:
Generic_INTERNAL_ERROR: Can not read value at 0 in block -1 in S3/XXXXXX/XXX/XXXX/XXXXX/date=20XX-XX-XX.parquet.

**To Reproduce**

Steps to reproduce the behavior:

1. Hudi Options like below:
Table ddl (_rt, _ro) was created automatically when enabling the hive sync tool.
![00a427c6f6f4d8e2a73b07ccf75bdf5](https://github.com/apache/hudi/assets/46934296/ca8fb5d0-2905-4292-8ae1-3309b23250a8)
record keys are strings, partition keys are just dates in yyyy-mm-dd
2.
![e3471b52ec345815027c1dd2a759cce](https://github.com/apache/hudi/assets/46934296/8d96a9ce-fc89-41a7-9d3c-4be5c826fe4d)

**Expected behavior**

I wish to keep my parquet size around 120 MB for each file (one partition) or two parquet files (around 64 MB).

**Environment Description**

* Hudi version : 0.13.0

* Spark version : 3.4.1

* Hive version : 0.13.1

* Hadoop version :

* Storage (HDFS/S3/GCS..) : S3

* Running on Docker? (yes/no) : NO

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the reported insert and inline-clustering workflow on Hudi 0.13.0 with Spark 3.4.1 and S3, using the configuration shown in the issue and date-partitioned data. Compare the generated Parquet sizes and test the clustered output through Athena. Done means isolating whether the small files or Athena read error is caused by configuration or a Hudi defect, with a documented resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.