Optimization whether to query Bootstrapped table using HoodieBootstrapRelation vs Sparks Parquet datasource
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
This has been discussed in [https://github.com/apache/hudi/pull/1702#discussion_r466317612]
As of now, while querying using *DataSource* we are checking if the table has been bootstrapped by the present of *bootstrap base path* in *hoodie.properties* file, and based on that query the table using *HoodieBootstrapRelation* vs *Spark Parquet Data Source*. However, there could be a scenario where all the files in the originally bootstrapped table have wither been *upserted/deleted* and thus have been fully bootstrapped and their data has been moved over to the target hoodie table. For such tables, we can start querying them using *Spark Parquet Data Source* which will be faster with all of spark's optimizations.
So, basically we a need a way to check if all of the files have been fully bootstrapped and moved over to the target location.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-1157
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-1265
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing DataSource table selection between HoodieBootstrapRelation and Spark Parquet Data Source, then review the linked pull request discussion and JIRA task for the intended bootstrap-state check. Done means tables whose bootstrapped files have all moved to the target location use Spark Parquet Data Source without changing other bootstrap queries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100