apache / apache/hudi

Add hash partition

Open
#15,803 1 comment 0 reactions 1 assignee Claimed by @yihua View on GitHub
engine:spark from-jira priority:high status:pr-available type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

It is often difficult to find an appropriate partition key in the existing data. Hash partitioning can easily solve this problem

When hash.partition.fields is specified and partition.fields contains _hoodie_hash_partition, a column named _hoodie_hash_partition will be added in this table as one of the partition key.

If predicates of hash.partition.fields appear in the query statement, the _hoodie_hash_partition = X predicate will be automatically added to the query statement for partition pruning.

 

The PR 7984  implements hash partitioning.

[https://github.com/apache/hudi/pull/7984]

 

How to use hash partition in spark data source can refer to hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala#testHashPartition

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-5810
- Type: New Feature
- Epic: https://issues.apache.org/jira/browse/HUDI-1297

---

## Comments

16/Feb/23 04:05;lvhu18@163.com;The PR 7984  implements hash partitioning.

https://github.com/apache/hudi/pull/7984

How to use hash partition in spark data source can refer to hudi/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala#testHashPartition;;;

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.