[Feature] Atomic schema evolution for chain tables
- Dominant language
- Java
- Stars
- 3.4k
- Forks
- 1.4k
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 396
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.
### Motivation
#### Background
Chain tables use three branches — `main`, `snapshot`, and `delta` — each maintaining its own independent schema log (`schema/schema-N`). The current documentation requires users to keep schemas consistent across all three branches manually (e.g. when setting up fallback branches or partition expiration options).
Today, a single logical `ALTER TABLE` on the base table only commits schema to **one** branch. There is no catalog-level coordination to update main, snapshot, and delta atomically.
#### Problem
Because each branch commits schema independently via `SchemaManager.commit()` → `FileIO.tryToWriteAtomic`, partial failures can leave the chain group in an inconsistent state:
1. **Immediate read failure**: `FallbackReadFileStoreTable.validateSchema()` / `ChainGroupReadTable.newScan()` checks row type and primary key consistency across branches. If e.g. `ADD COLUMN` succeeds on `main` but fails on `snapshot`/`delta`, the entire chain table becomes unreadable at scan planning time.
2. **Operational burden**: Users must remember to run the same DDL three times (main + snapshot + delta), which is error-prone and not enforced by the catalog.
Example failure after partial `ADD COLUMN`:
```
main latest: schema-11, fields = [a, b, c]
snapshot latest: schema-10, fields = [a, b]
delta latest: schema-10, fields = [a, b]
```
→ chain reader fails with "does not have the same row type" before any data file is read.
#### Current behavior (as of master)
- `AbstractCatalog.alterTable` delegates to `alterTableImpl` for a single identifier; no chain-table-aware multi-branch sync.
- `FileStoreTableFactory.createChainTable` loads snapshot/delta schemas from their respective branch-local `SchemaManager`.
- Docs explicitly state: *"Chain table should ensure that the schema of each branch is consistent."*
- No rollback if multi-branch DDL is attempted manually and fails midway.
### Solution
1. **Main branch schema log becomes the single canonical source** for shared fields / field IDs / PK / partition keys.
2. **One atomic write** to `main/schema/schema-N` publishes the schema for the entire chain group.
3. **Branch-specific options** (e.g. different `bucket` per branch) are stored in the canonical main schema and materialized per branch at read/write time.
4. **Legacy schemas before a cutover point** continue to be resolved from branch-local schema files for backward compatibility.
#### Example: canonical main schema with per-branch options
Shared columns / PK / partition keys live in the top-level `TableSchema`. Per-branch effective options are stored as JSON strings in `options` (exact key names TBD):
```json
{
"id": 25,
"fields": [
{"id": 0, "name": "id", "type": "BIGINT NOT NULL"},
{"id": 1, "name": "dt", "type": "STRING NOT NULL"}
],
"partitionKeys": ["dt"],
"primaryKeys": ["id", "dt"],
"options": {
"chain-table.enabled": "true",
"scan.fallback-snapshot-branch": "snapshot",
"scan.fallback-delta-branch": "delta",
"bucket": "64",
"bucket-key": "id",
"chain-table.member-options.snapshot": "{\"bucket\":\"32\",\"bucket-key\":\"id\"}",
"chain-table.member-options.delta": "{\"bucket\":\"128\",\"bucket-key\":\"id\"}"
}
}
```
At resolve time, each branch gets a normal `TableSchema` with the **same** `id`, fields, PK, and partition keys, but different effective options:
| Branch | Effective `bucket` | Source |
|----------|-------------------|--------|
| `main` | 64 | top-level `options` |
| `snapshot` | 32 | `member-options.snapshot` |
| `delta` | 128 | `member-options.delta` |
A base-table `ALTER TABLE ... SET TBLPROPERTIES ('bucket' = '96')` would update all three members in one canonical commit. An explicit `ALTER TABLE t$branch_delta SET TBLPROPERTIES ('bucket' = '256')` would only update `member-options.delta`, still via a single main schema write.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading AbstractCatalog.alterTable and alterTableImpl, then trace SchemaManager.commit and FileStoreTableFactory.createChainTable. Review FallbackReadFileStoreTable.validateSchema and ChainGroupReadTable.newScan to understand the consistency checks. Done means the chain group's schema evolution is coordinated through the canonical main schema while preserving legacy branch-local resolution.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100