apache / apache/hudi

Addition of virtual _hoodie_commit_completion_time column which is used for Incremental Queries

Open
#14,036 0 comments 0 reactions 0 assignees View on GitHub
type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Bug Description

**What happened:**
Filtering for Hudi incremental queries on tables with version >= 6 depends on commit completion time which is not present in the output for the users. Filtering with requested times (currently outputted) gives inconsistent and wrong results. It is difficult for the users to give proper times as part of the filter if it is not outputted.

**What you expected:**
Implement a completion time virtual column (_hoodie_commit_completion_time) for Hudi incremental queries on tables with version >= 6. This virtual column is dynamically added during query execution and provides completion time semantics for filtering and display.

**Steps to reproduce:**
1. Create a Hudi Table with incremental query support
2. Performing Incremental Query with requested times (which is outputted currently) gives inconsistent and wrong results.

### Environment

**Hudi version:** 1.1
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**

### Logs and Stack Trace

_No response_

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by identifying the Hudi 1.1 implementation for incremental queries and how requested times are currently exposed; the query engine is not specified. Reproduce the inconsistent filtering, then verify that tables with version >= 6 dynamically expose _hoodie_commit_completion_time with completion-time semantics for both filtering and display.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.