apache / apache/hudi

Support writes using spark dataframe end to end

Open
#16,846 1 comment 0 reactions 1 assignee Claimed by @nsivabalan View on GitHub
from-jira priority:critical type:improvement type:story
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

We wanted to support writes using spark end to end using dataframe w/o converting them to avro record.

 

This opens up lot of opportunities for Hudi 
* This will place Hudi close to direct parquet writes for straight forward immutable use-cases. Also for mutable use-cases, it will increase
* For mutable use-cases, we are anticipating 10 to 20% improvement over rdd based write client impl. 
* We can leverage spark optimizations which can kick in only with dataframe. 
* Rapids, vectorized reading etc can speed up writes with Hudi once we move to end to end data frame writes. 

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-9019
- Type: Improvement
- Fix version(s):
- 1.1.0

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.