apache / apache/hudi

Optimization whether to query Bootstrapped table using HoodieBootstrapRelation vs Sparks Parquet datasource

Open
#14,634 0 comments 0 reactions 0 assignees View on GitHub
component:bootstrap from-jira priority:blocker type:devtask
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

This has been discussed in [https://github.com/apache/hudi/pull/1702#discussion_r466317612]

As of now, while querying using *DataSource* we are checking if the table has been bootstrapped by the present of *bootstrap base path* in *hoodie.properties* file, and based on that query the table using *HoodieBootstrapRelation*  vs *Spark Parquet Data Source*. However, there could be a scenario where all the files in the originally bootstrapped table have wither been *upserted/deleted* and thus have been fully bootstrapped and their data has been moved over to the target hoodie table. For such tables, we can start querying them using *Spark Parquet Data Source* which will be faster with all of spark's optimizations.

So, basically we a need a way to check if all of the files have been fully bootstrapped and moved over to the target location.

## JIRA info

- Link: https://issues.apache.org/jira/browse/HUDI-1157
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-1265

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing DataSource table selection between HoodieBootstrapRelation and Spark Parquet Data Source, then review the linked pull request discussion and JIRA task for the intended bootstrap-state check. Done means tables whose bootstrapped files have all moved to the target location use Spark Parquet Data Source without changing other bootstrap queries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.