DataFusion: paimon_incremental_query table function
- Dominant language
- Rust
- Stars
- 197
- Forks
- 92
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 91
Description
### Describe the feature
Expose batch incremental reads in DataFusion SQL via a table function, roughly:
```sql
SELECT * FROM paimon_incremental_query(
'table_name',
start_snapshot_exclusive,
end_snapshot_inclusive
[, 'auto' | 'delta' | 'changelog' | 'diff']
);
```
Also support the `$audit_log` style suffix so row kinds show up as a first column when needed.
### Motivation
The Rust core incremental API is useful, but most people will want to poke at this from SQL. DataFusion should only parse args and adapt the execution plan — snapshot planning stays in `paimon` core.
### Depends on
Core incremental Delta / Changelog / Diff support landing first (or the PR can sit on top of that stack).
### Out of scope
- Reimplementing snapshot planning inside the DataFusion crate
- Lookup/compact changelog generation
Contributor guide
Research direction
Start in the DataFusion crate by reading the existing table-function and execution-plan adaptation entry points, then inspect the Rust core incremental API and its Delta, Changelog, and Diff support. The work is done when SQL accepts the stated snapshot and mode arguments, preserves planning in paimon core, and supports the $audit_log suffix without implementing lookup or compact changelog generation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- data-engineering, database
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100