apache / apache/pinot

Schema backward-compatibility: first-class support for column deletion

Open
#18,808 8 comments 2 reactions 1 assignee Claimed by @sourabh-27 View on GitHub
configuration enhancement
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

### Feature request

Schema updates currently treat **column deletion as backward-incompatible**. Removing a column from a table schema is rejected unless the operator passes `force=true`, which bypasses all validation rather than handling the deletion safely.

### Current behavior

- `Schema.isBackwardCompatibleWith()` (`pinot-spi/src/main/java/org/apache/pinot/spi/data/Schema.java`) returns `false` if any column present in the old schema is missing from the new one.
- `PinotHelixResourceManager.updateSchema()` throws `SchemaBackwardIncompatibleException` → HTTP 400 _"Only allow adding new columns"_, unless `force=true`.
- With `force=true` the check is skipped with only a warning log. The result is an inconsistent state:
- Existing segments still physically contain the deleted column.
- New segments don't have it.
- Queries referencing the deleted column **silently return null** (`PinotSegmentColumnReaderFactory`) rather than erroring or being cleaned up.
- There is no `allowDeletedColumn` / managed-deletion config.

### Proposed

Make column deletion a first-class, backward-compatible operation instead of an all-or-nothing `force` override:

1. Allow column removal through schema update without `force`, behind explicit intent (e.g. a dedicated flag or validation path that distinguishes "drop column" from accidental omission).
2. Define + document segment behavior for deleted columns — reclaim the column on segment reload, and decide query semantics (ignore vs. error on reference).
3. Preserve the existing guardrails (primary-key changes, type changes) — only column *removal* is relaxed.

### Notes

- Backward-compatibility / rolling-upgrade sensitive: mixed-version brokers/servers must agree on how a queried-but-deleted column behaves.
- Related validation: `FieldSpec.isBackwardCompatibleWith()`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.