Addition of virtual _hoodie_commit_completion_time column which is used for Incremental Queries
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Bug Description
**What happened:**
Filtering for Hudi incremental queries on tables with version >= 6 depends on commit completion time which is not present in the output for the users. Filtering with requested times (currently outputted) gives inconsistent and wrong results. It is difficult for the users to give proper times as part of the filter if it is not outputted.
**What you expected:**
Implement a completion time virtual column (_hoodie_commit_completion_time) for Hudi incremental queries on tables with version >= 6. This virtual column is dynamically added during query execution and provides completion time semantics for filtering and display.
**Steps to reproduce:**
1. Create a Hudi Table with incremental query support
2. Performing Incremental Query with requested times (which is outputted currently) gives inconsistent and wrong results.
### Environment
**Hudi version:** 1.1
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by identifying the Hudi 1.1 implementation for incremental queries and how requested times are currently exposed; the query engine is not specified. Reproduce the inconsistent filtering, then verify that tables with version >= 6 dynamically expose _hoodie_commit_completion_time with completion-time semantics for both filtering and display.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100