[Feature] [core] Support arbitrary time granularity for chain table delta computation
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
Users need chain tables with minute-level granularity for scenarios like:
- IoT data ingestion with minute-level partitions
- Real-time monitoring systems
- Incremental processing with fine-grained time windows
Example configuration:
```
PARTITIONED BY (`dt` STRING, `hr_min` STRING)
TBLPROPERTIES (
'partition.timestamp-pattern' = '$dt $hr_min:00',
'partition.timestamp-formatter' = 'yyyyMMdd HH:mm:ss',
'chain-table.chain-partition-keys' = 'dt,hr_min'
)
```
or
```
PARTITIONED BY (`dt` STRING, `hr` STRING, `min` STRING)
TBLPROPERTIES (
'partition.timestamp-pattern' = '$dt $hr:$min:00',
'partition.timestamp-formatter' = 'yyyyMMdd HH:mm:ss',
'chain-table.chain-partition-keys' = 'dt,hr_min'
)
```
### Solution
Introduce `PartitionTimeResolver` to dynamically infer the time step from the `partition.timestamp-pattern` and `partition.timestamp-formatter` configuration.
### Anything else?
Refactor `ChainTableUtils.calPartValues`, support pattern like $y$m$d
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at ChainTableUtils.calPartValues and trace how partition.timestamp-pattern, partition.timestamp-formatter, and chain-table.chain-partition-keys are currently used. Read the proposed PartitionTimeResolver design and verify how minute-level and $y$m$d patterns should infer their time step. Done means chain table delta computation supports arbitrary configured time granularity and the existing examples continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100