Support writes using spark dataframe end to end
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
We wanted to support writes using spark end to end using dataframe w/o converting them to avro record.
This opens up lot of opportunities for Hudi
* This will place Hudi close to direct parquet writes for straight forward immutable use-cases. Also for mutable use-cases, it will increase
* For mutable use-cases, we are anticipating 10 to 20% improvement over rdd based write client impl.
* We can leverage spark optimizations which can kick in only with dataframe.
* Rapids, vectorized reading etc can speed up writes with Hudi once we move to end to end data frame writes.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9019
- Type: Improvement
- Fix version(s):
- 1.1.0
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.