[SUPPORT] Datasource incremental subsequent read same as first read
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
hudi 0.12.2
spark 3.2.1
-----------------
Once an incremental read is made, all subsequent read on the table will remain the same. Likely something about incremental is cached.
NOTE: this behavior does not happens when loading the table by path (instead of with database/table)
```scala
// query 1
spark.read.format("hudi")
.option("hoodie.metadata.enable","true")
.table("database.hudi_table").count()
// 1000
// query 2
spark.read.format("hudi")
.option("hoodie.metadata.enable","true")
.option("hoodie.datasource.query.type","incremental")
.option("hoodie.datasource.read.begin.instanttime","20230203191804078")
.table("database.hudi_table").count()
// 200
// query 3
spark.read.format("hudi")
.option("hoodie.metadata.enable","true")
.table("database.hudi_table").count()
// 200 should be 1000
// query 4 after restarting the spark shell
spark.read.format("hudi")
.option("hoodie.metadata.enable","true")
.table("database.hudi_table").count()
// 1000
```
Also weird, the number of tasks is way smaller if query 2 is run in a fresh spark session.
- query 1 before query 2, then 25k tasks
- query 2 without query 1, then 17 tasks
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the four Spark reads in Hudi 0.12.2 with Spark 3.2.1, comparing database/table loading with path loading and recording the task counts in fresh and reused sessions. Done means a later non-incremental table read returns 1000 rather than the incremental result of 200, with the task-count difference understood and resolved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, scala
- Domain
- data-engineering, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100