apache / apache/paimon

[Feature] [core] Support arbitrary time granularity for chain table delta computation

Open
#8,184 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

Users need chain tables with minute-level granularity for scenarios like:
- IoT data ingestion with minute-level partitions
- Real-time monitoring systems
- Incremental processing with fine-grained time windows

Example configuration:
```
PARTITIONED BY (`dt` STRING, `hr_min` STRING)
TBLPROPERTIES (
'partition.timestamp-pattern' = '$dt $hr_min:00',
'partition.timestamp-formatter' = 'yyyyMMdd HH:mm:ss',
'chain-table.chain-partition-keys' = 'dt,hr_min'
)
```
or
```
PARTITIONED BY (`dt` STRING, `hr` STRING, `min` STRING)
TBLPROPERTIES (
'partition.timestamp-pattern' = '$dt $hr:$min:00',
'partition.timestamp-formatter' = 'yyyyMMdd HH:mm:ss',
'chain-table.chain-partition-keys' = 'dt,hr_min'
)
```

### Solution

Introduce `PartitionTimeResolver` to dynamically infer the time step from the `partition.timestamp-pattern` and `partition.timestamp-formatter` configuration.

### Anything else?

Refactor `ChainTableUtils.calPartValues`, support pattern like $y$m$d

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at ChainTableUtils.calPartValues and trace how partition.timestamp-pattern, partition.timestamp-formatter, and chain-table.chain-partition-keys are currently used. Read the proposed PartitionTimeResolver design and verify how minute-level and $y$m$d patterns should infer their time step. Done means chain table delta computation supports arbitrary configured time granularity and the existing examples continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.