apache / apache/hudi

Improve usability of bulk insert with insert overwrite operations in Spark Datasource

Open
#16,245 1 comment 0 reactions 0 assignees View on GitHub
from-jira priority:high type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

Currently for using bulk insert with insert overwrite operations in Spark Datasource, user would currently have to set `hoodie.bulkinsert.overwrite.operation.type` to insert_overwrite_table(OVERWRITE) or insert_overwrite(APPEND) and use the corresponding save modes. Since this is an internal config, it should not be exposed. The jira aims to find an easier way to support the feature for the users through a new config or a different config altogether.
One idea is to deprecate hoodie.spark.sql.insert.into.operation and create a new config which can be shared by both sql and datasource.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-6889
- Type: Bug

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing Spark Datasource bulk-insert handling for insert overwrite operations and compare the current hoodie.bulkinsert.overwrite.operation.type and hoodie.spark.sql.insert.into.operation settings with their save modes. Done means users can select the overwrite behavior through an easier shared or replacement configuration without exposing the internal setting.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.