[SUPPORT] How to reduce the bucket number for each partition
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
At present, spark provides to increase bucket numbers automatically.
But in the case of partitioning, the data volume of each partition is not necessarily balanced.
The initially specified bucket number is generally calculated according to the maximum partition, resulting in too many small partitions with the same number.
It cannot reduce the bucket number of small partitions at present.
**Environment Description**
* Hudi version : 0.12.1
* Spark version : 3.1.3
* Hive version : 3.1.0
* Hadoop version : 3.1.1
* Storage (HDFS/S3/GCS..) : HDFS
* Running on Docker? (yes/no) : no
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue provides no source files, tests, or entry points. Start by investigating how Hudi handles Spark bucket allocation for partitioned data, using the stated Hudi, Spark, Hive, Hadoop, and HDFS versions; done means smaller partitions can use fewer buckets without disrupting partitioned writes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100