Improve usability of bulk insert with insert overwrite operations in Spark Datasource
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
Currently for using bulk insert with insert overwrite operations in Spark Datasource, user would currently have to set `hoodie.bulkinsert.overwrite.operation.type` to insert_overwrite_table(OVERWRITE) or insert_overwrite(APPEND) and use the corresponding save modes. Since this is an internal config, it should not be exposed. The jira aims to find an easier way to support the feature for the users through a new config or a different config altogether.
One idea is to deprecate hoodie.spark.sql.insert.into.operation and create a new config which can be shared by both sql and datasource.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6889
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing Spark Datasource bulk-insert handling for insert overwrite operations and compare the current hoodie.bulkinsert.overwrite.operation.type and hoodie.spark.sql.insert.into.operation settings with their save modes. Done means users can select the overwrite behavior through an easier shared or replacement configuration without exposing the internal setting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100