apache / apache/paimon

[Feature] Ignore tag.creation-delay with a new spark-sql option when calling trigger_tag_automatic_creation

Open
#6,450 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

The procedure `trigger_tag_automatic_creation` was designed to force creating an auto-tag after the right tag time.
For example, if called after hour 0, an auto tag should be created for a daily auto-tagged table which has not been auto-tagged with the name of yesterday.

But it doesn't work under some conditions
* No snapshot exits since the auto-tag time;
* The calling time is before the delay time.

The reason is that
1. Spark is not like Flink which could commit an empty message and make a new snapshot, so when triggered there may be not an appropriate snapshot for tag.
2. Even if there is a snapshot since the auto-tag time, the TagAutoCreation may still refuse to create since it needs a snapshot after the delay time.

For example
1. T is daily tag table with creation-delay=10m
2. spark-sql call this procedure at 2025-10-22T00:02:00,
3. if there is no snapshot between 00:00:0 ~ 00:02:00, creating no auto-tag
4. if there is a snapshot at 00:01:00, but it's earlier than the delay time, still creating no auto-tag.

### Solution
1. Like Flink, spark will also commit empty message to make sure a snapshot when calling trigger procedure.
2. Set up a new spark-sql conf option, like
```
set `spark.paimon.trigger-tag-auto-creation-ignore-delay` = true;
```
if the option is true, creat an auto tag even if the snapshot is earlier than the delay.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Spark SQL procedure trigger_tag_automatic_creation and inspect the existing Flink empty-commit behavior, then trace TagAutoCreation's creation-delay check. The work is done when Spark creates a snapshot when needed and the spark.paimon.trigger-tag-auto-creation-ignore-delay option allows tag creation before the delay when enabled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.