apache / apache/hudi

[SUPPORT] Datasource incremental subsequent read same as first read

Open
#7,846 7 comments 0 reactions 0 assignees View on GitHub
area:incr-processing engine:spark priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

hudi 0.12.2
spark 3.2.1
-----------------

Once an incremental read is made, all subsequent read on the table will remain the same. Likely something about incremental is cached.
NOTE: this behavior does not happens when loading the table by path (instead of with database/table)

```scala
// query 1
spark.read.format("hudi")
.option("hoodie.metadata.enable","true")
.table("database.hudi_table").count()
// 1000

// query 2
spark.read.format("hudi")
.option("hoodie.metadata.enable","true")
.option("hoodie.datasource.query.type","incremental")
.option("hoodie.datasource.read.begin.instanttime","20230203191804078")
.table("database.hudi_table").count()
// 200

// query 3
spark.read.format("hudi")
.option("hoodie.metadata.enable","true")
.table("database.hudi_table").count()
// 200 should be 1000

// query 4 after restarting the spark shell
spark.read.format("hudi")
.option("hoodie.metadata.enable","true")
.table("database.hudi_table").count()
// 1000
```

Also weird, the number of tasks is way smaller if query 2 is run in a fresh spark session.
- query 1 before query 2, then 25k tasks
- query 2 without query 1, then 17 tasks

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the four Spark reads in Hudi 0.12.2 with Spark 3.2.1, comparing database/table loading with path loading and recording the task counts in fresh and reused sessions. Done means a later non-incremental table read returns 1000 rather than the incremental result of 200, with the task-count difference understood and resolved.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, scala
Domain
data-engineering, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.