apache / apache/paimon

[Feature] Atomic schema evolution for chain tables

Open
#8,810 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Motivation

#### Background

Chain tables use three branches — `main`, `snapshot`, and `delta` — each maintaining its own independent schema log (`schema/schema-N`). The current documentation requires users to keep schemas consistent across all three branches manually (e.g. when setting up fallback branches or partition expiration options).

Today, a single logical `ALTER TABLE` on the base table only commits schema to **one** branch. There is no catalog-level coordination to update main, snapshot, and delta atomically.

#### Problem

Because each branch commits schema independently via `SchemaManager.commit()` → `FileIO.tryToWriteAtomic`, partial failures can leave the chain group in an inconsistent state:

1. **Immediate read failure**: `FallbackReadFileStoreTable.validateSchema()` / `ChainGroupReadTable.newScan()` checks row type and primary key consistency across branches. If e.g. `ADD COLUMN` succeeds on `main` but fails on `snapshot`/`delta`, the entire chain table becomes unreadable at scan planning time.

2. **Operational burden**: Users must remember to run the same DDL three times (main + snapshot + delta), which is error-prone and not enforced by the catalog.

Example failure after partial `ADD COLUMN`:

```
main latest: schema-11, fields = [a, b, c]
snapshot latest: schema-10, fields = [a, b]
delta latest: schema-10, fields = [a, b]
```

→ chain reader fails with "does not have the same row type" before any data file is read.

#### Current behavior (as of master)

- `AbstractCatalog.alterTable` delegates to `alterTableImpl` for a single identifier; no chain-table-aware multi-branch sync.
- `FileStoreTableFactory.createChainTable` loads snapshot/delta schemas from their respective branch-local `SchemaManager`.
- Docs explicitly state: *"Chain table should ensure that the schema of each branch is consistent."*
- No rollback if multi-branch DDL is attempted manually and fails midway.

### Solution

1. **Main branch schema log becomes the single canonical source** for shared fields / field IDs / PK / partition keys.
2. **One atomic write** to `main/schema/schema-N` publishes the schema for the entire chain group.
3. **Branch-specific options** (e.g. different `bucket` per branch) are stored in the canonical main schema and materialized per branch at read/write time.
4. **Legacy schemas before a cutover point** continue to be resolved from branch-local schema files for backward compatibility.

#### Example: canonical main schema with per-branch options

Shared columns / PK / partition keys live in the top-level `TableSchema`. Per-branch effective options are stored as JSON strings in `options` (exact key names TBD):

```json
{
"id": 25,
"fields": [
{"id": 0, "name": "id", "type": "BIGINT NOT NULL"},
{"id": 1, "name": "dt", "type": "STRING NOT NULL"}
],
"partitionKeys": ["dt"],
"primaryKeys": ["id", "dt"],
"options": {
"chain-table.enabled": "true",
"scan.fallback-snapshot-branch": "snapshot",
"scan.fallback-delta-branch": "delta",

"bucket": "64",
"bucket-key": "id",

"chain-table.member-options.snapshot": "{\"bucket\":\"32\",\"bucket-key\":\"id\"}",
"chain-table.member-options.delta": "{\"bucket\":\"128\",\"bucket-key\":\"id\"}"
}
}
```

At resolve time, each branch gets a normal `TableSchema` with the **same** `id`, fields, PK, and partition keys, but different effective options:

| Branch | Effective `bucket` | Source |
|----------|-------------------|--------|
| `main` | 64 | top-level `options` |
| `snapshot` | 32 | `member-options.snapshot` |
| `delta` | 128 | `member-options.delta` |

A base-table `ALTER TABLE ... SET TBLPROPERTIES ('bucket' = '96')` would update all three members in one canonical commit. An explicit `ALTER TABLE t$branch_delta SET TBLPROPERTIES ('bucket' = '256')` would only update `member-options.delta`, still via a single main schema write.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading AbstractCatalog.alterTable and alterTableImpl, then trace SchemaManager.commit and FileStoreTableFactory.createChainTable. Review FallbackReadFileStoreTable.validateSchema and ChainGroupReadTable.newScan to understand the consistency checks. Done means the chain group's schema evolution is coordinated through the canonical main schema while preserving legacy branch-local resolution.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.