apache / apache/paimon

[Bug] Chain table Delta commits do not expire Snapshot branch snapshots after partition expiration

Open
#9,595 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Minimal reproduce step

1. Create a non-write-only Chain Table with partition expiration disabled initially.

2. Write an initial baseline to the Snapshot branch using an old partition, for example `current_date - 90 days`.

3. Write data to the Delta branch using another old partition, for example `current_date - 65 days`.

4. Run Snapshot compaction for that Delta partition:

```sql
CALL sys.compact_chain_table(
table => 'test.chain_table',
partition => 'dt='
);
```

This creates the second anchor partition and another metadata snapshot on the Snapshot branch.

5. Write another Delta partition, for example `current_date - 40 days`, and run Snapshot compaction again:

```sql
CALL sys.compact_chain_table(
table => 'test.chain_table',
partition => 'dt='
);
```

The Snapshot branch should now contain three anchor partitions and multiple metadata snapshots.

6. Enable Chain Table partition expiration and aggressive snapshot retention:

```sql
ALTER TABLE chain_table SET TBLPROPERTIES (
'write-only' = 'false',
'partition.expiration-time' = '30 d',
'end-input.check-partition-expire' = 'true',
'snapshot.num-retained.min' = '1',
'snapshot.num-retained.max' = '1',
'snapshot.time-retained' = '0 s',
'snapshot.expire.execution-mode' = 'sync'
);
```

7. From this point onward, write only to the Delta branch. Use a non-expired partition, for example `current_date - 10 days`, so that the final bounded commit triggers partition expiration:

```sql
INSERT INTO `chain_table$branch_delta`
PARTITION (dt='')
VALUES (1, 400, 'partition-expiration-trigger');
```

8. Check the Snapshot branch partitions:

```sql
SHOW PARTITIONS `chain_table$branch_snapshot`;
```

The expired Snapshot anchor partitions should have been deleted, while the newest required anchor should remain.

9. Check the Snapshot branch metadata history:

```sql
SELECT snapshot_id, commit_kind
FROM `chain_table$branch_snapshot$snapshots`
ORDER BY snapshot_id;
```

10. Check whether the data files referenced only by the expired Snapshot-branch snapshots have been deleted.

### What doesn't meet your expectations?

`ChainTablePartitionExpire` deletes expired partitions from the Snapshot branch. The partition deletion commits a new metadata snapshot to that branch.

However, the expiration runnable associated with the triggering Delta commit expires snapshots only on the Delta branch. It does not run snapshot expiration on the Snapshot branch after the partition deletion.

Consequently, the Snapshot branch still contains multiple metadata snapshots even though the table is configured with:

```text
snapshot.num-retained.min = 1
snapshot.num-retained.max = 1
snapshot.time-retained = 0s
```

Data files referenced by those old snapshots also cannot be reclaimed until the corresponding Snapshot-branch snapshots are expired.

This is particularly problematic for the normal Chain Table workflow:

1. Periodic `compact_chain_table` operations create Snapshot anchors.
2. Subsequent ingestion writes only to the Delta branch.
3. Partition expiration deletes old Snapshot partitions.
4. No later direct Snapshot-branch commit is guaranteed to trigger snapshot expiration for that branch.

**Expected behavior:**

After Chain Table partition expiration commits partition deletions to the Snapshot branch, automatic maintenance should expire Snapshot-branch snapshots according to the configured retention policy.

With the configuration above, the Snapshot branch should retain only its latest metadata snapshot, and files referenced exclusively by expired snapshots should be deleted.

### Anything else?

Relevant code:

- [`AbstractFileStoreTable.newExpireRunnable()`](https://github.com/apache/paimon/blob/85c8d601acd85c1a69fe036b451b28b595e1edad/paimon-core/src/main/java/org/apache/paimon/table/AbstractFileStoreTable.java#L500) creates expiration operations only for the current table or branch.
- [`ChainTablePartitionExpire`](https://github.com/apache/paimon/blob/85c8d601acd85c1a69fe036b451b28b595e1edad/paimon-core/src/main/java/org/apache/paimon/operation/ChainTablePartitionExpire.java#L336) commits Snapshot-branch partition deletions without expiring that branch's snapshots.
- Original Chain Table partition-expiration PR: [#7643](https://github.com/apache/paimon/pull/7643).

**Suggested direction:**

Keep `ChainTablePartitionExpire` responsible for selecting and deleting expired partitions. Extend the normal expiration runnable for a Chain Delta table so that it expires:

1. Snapshots on the current Delta branch.
2. Snapshots on the configured Snapshot branch.

This issue concerns automatic maintenance behavior and does not require a separate public `expire_chain_snapshots` action.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Read AbstractFileStoreTable.newExpireRunnable() and ChainTablePartitionExpire, then run the minimal SQL reproduction to follow the Delta-triggered maintenance path. Ensure the existing partition deletion is followed by snapshot expiration on the Snapshot branch as well as the Delta branch, and verify that retention leaves only the configured latest snapshot and permits exclusive files to be removed.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.