apache / apache/datafusion-comet

Deepen v2-checkpoint regression coverage for the native Delta scan

Open
#5,660 0 comments 0 reactions 0 assignees View on GitHub
requires-triage
Dominant language
Scala
Stars
1.3k
Forks
373
Avg merge
2d 4h
Merged PRs (30d)
198

Description

### What is the problem the feature request solves?

Follow-up from the native Delta Lake scan review (#5365). The v2-checkpoint test performs several commits and reads immediately, so it does not establish that the read reconstructs the snapshot from the checkpoint rather than cached state. Coverage should verify actual checkpoint artifacts, include a sidecar-backed case, invalidate the Delta log cache, and reopen the snapshot before comparing Spark and native results. A post-checkpoint append plus a time-travel read would exercise the boundary between snapshot reconstruction and native table scanning.

### Describe the potential solution

No response

### Additional context

Raised in the #5365 review discussion.

Contributor guide

Open the contributing guide

Research direction

Start with the existing v2-checkpoint regression test and review the native Delta scan coverage discussed in #5365. Add cases that use real checkpoint artifacts, including a sidecar-backed checkpoint, then invalidate the Delta log cache and reopen the snapshot before comparing Spark and native results. A post-checkpoint append and time-travel read should verify the snapshot boundary.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, databases, testing
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.