apache / apache/paimon

[Feature] Support nearest-previous partition fallback for chain table when queried partition does not exist

Open
#8,779 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Currently, when querying a Paimon chain table table (e.g., time-series data partitioned by dt), if the queried partition does not exist, the query returns empty results. This is the default behavior.

In some scenarios , users expect the query to transparently fall back to the nearest previous existing partition rather than returning empty data.

### Solution

1.If the queried partition does not exist in the table
2.The system automatically finds the nearest previous partition (the largest partition value that is smaller than the queried one)
3.Returns data from that partition instead of empty results

## example
Table partitions: `[2024-07-01, 2024-07-02, 2024-07-05]`
snapshot paritions : `[2024-07-01]`
delta partitionss : `[2024-07-02, 2024-07-05]`

Query1 :
`SELECT * FROM table WHERE dt = '2024-07-04'`
Current behavior: returns empty
With feature enabled: returns data from `dt = '2024-07-02'` , and the value of dt column replaced with `2024-07-04`

Query2 :
`SELECT * FROM table WHERE dt = '2024-07-10'`
Current behavior: returns empty
With feature enabled: returns data from `dt = '2024-07-10' `, and the value of dt column replaced with `2024-07-10`

only work for query like : `partition column` = 'some value'
not work for query like : `partition column` <='some value' or `partition column` in ('some value')

### Anything else?

I would like to ask whether this feature is considered reasonable from the project's design perspective. If the maintainers agree that this is a valuable addition and the proposed approach is acceptable, I have already implemented the core logic locally and am ready to clean up the code and submit a PR.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue does not name implementation files, entry points, or tests. Start by locating the chain-table partition lookup and query handling for equality predicates, then verify the behavior for missing partitions, nearest-previous selection, and replacement of the returned partition value. Done means the feature's scope and design are accepted and covered by tests, including the stated exclusions for <= and IN predicates.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.