[Improvement] Support primary key on Iceberg tables via identifier-field-ids
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
The Iceberg catalog (`catalog-lakehouse-iceberg`) currently rejects any
`Index` supplied at table creation. `IcebergCatalogOperations#createTable`
unconditionally throws when `indexes != null && indexes.length > 0`, with the
message *"Iceberg does not support indexes"*.
However, Iceberg V2 tables natively support a primary key concept via the
schema-level `identifier-field-ids`, which is what engines such as Flink CDC
and the Iceberg Flink connector rely on for upsert / equality-delete writes
(see `FlinkSchemaUtil.freshIdentifierFieldIds` in apache/iceberg). Because
Gravitino refuses the `PRIMARY_KEY` index up-front, users cannot:
1. Express a primary key on an Iceberg table through the Gravitino API or
any engine that goes through Gravitino (e.g. Spark/Flink via the
Gravitino catalog).
2. Round-trip a primary key created by another engine: even if the underlying
Iceberg table already carries `identifier-field-ids`, `IcebergTable`'s
load path (`fromIcebergTable`) drops them, so `Table#index()` always
returns empty for Iceberg.
The Paimon catalog already handles the analogous case (Paimon primary keys ↔
Gravitino `PRIMARY_KEY` index) in both directions — see
`PaimonTable#constructIndexesFromPrimaryKeys` and
`GravitinoToPaimonTableConverter`. The Iceberg catalog should align with that
behavior.
### How should we improve?
Wire the Gravitino `PRIMARY_KEY` index through to / from Iceberg's
`identifier-field-ids`, forming a create/load round-trip closure:
- **Forward (create):** in `ConvertUtil#toIcebergSchema`, after building the
Iceberg `Schema`, set `identifier-field-ids` from the field ids of the
columns referenced by the single `PRIMARY_KEY` index, mirroring
`FlinkSchemaUtil.freshIdentifierFieldIds`. Iceberg requires identifier
fields to be `required` (NOT NULL); rely on Iceberg's own `Schema`
constructor to enforce that constraint (do not silently promote
nullability).
- **Reverse (load):** in `IcebergTable#fromIcebergTable`, reconstruct a
`PRIMARY_KEY` index from `Schema#identifierFieldIds()`, ordered by the
column position in the schema (Iceberg identifier fields are an unordered
set, so an explicit ordering is needed for determinism). Use a fixed
synthetic index name `ICEBERG_PRIMARY_KEY_INDEX`, analogous to
Paimon's `PAIMON_PRIMARY_KEY_INDEX`.
- **Validation:** replace the blanket "indexes not supported" rejection in
`IcebergCatalogOperations#createTable` with a focused check that allows at
most one `PRIMARY_KEY` index, with at least one column, and rejects nested
column references (`fieldName.length == 1`). All other `Index` shapes
(multiple indexes, non-`PRIMARY_KEY` types, nested columns) continue to be
rejected with a clear message.
Out of scope for this issue:
- Altering the primary key via `TableChange` — Iceberg's
`UpdateSchema#setIdentifierFields` exists, but PK evolution has additional
semantics (NOT NULL promotion of existing columns, interaction with
equality deletes) that warrant a separate discussion.
- Changes to the Iceberg REST server module (`iceberg-rest-server`).
I have a working patch with unit tests covering forward mapping, reverse
mapping, full round-trip, and validation rejection cases (multiple indexes,
non-PK index type, nested PK column, empty PK column list). Happy to open a
PR linked to this issue once it is accepted.
Contributor guide
Research direction
Start with IcebergCatalogOperations#createTable, ConvertUtil#toIcebergSchema, and IcebergTable#fromIcebergTable, then compare PaimonTable#constructIndexesFromPrimaryKeys and GravitinoToPaimonTableConverter. Run the Iceberg catalog unit tests covering forward mapping, reverse mapping, round trips, and validation rejection cases. Done means PRIMARY_KEY metadata maps in both directions while unsupported index shapes remain rejected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100