apache / apache/gravitino

[Improvement] Support primary key on Iceberg tables via identifier-field-ids

Open
#11,753 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

The Iceberg catalog (`catalog-lakehouse-iceberg`) currently rejects any
`Index` supplied at table creation. `IcebergCatalogOperations#createTable`
unconditionally throws when `indexes != null && indexes.length > 0`, with the
message *"Iceberg does not support indexes"*.

However, Iceberg V2 tables natively support a primary key concept via the
schema-level `identifier-field-ids`, which is what engines such as Flink CDC
and the Iceberg Flink connector rely on for upsert / equality-delete writes
(see `FlinkSchemaUtil.freshIdentifierFieldIds` in apache/iceberg). Because
Gravitino refuses the `PRIMARY_KEY` index up-front, users cannot:

1. Express a primary key on an Iceberg table through the Gravitino API or
any engine that goes through Gravitino (e.g. Spark/Flink via the
Gravitino catalog).
2. Round-trip a primary key created by another engine: even if the underlying
Iceberg table already carries `identifier-field-ids`, `IcebergTable`'s
load path (`fromIcebergTable`) drops them, so `Table#index()` always
returns empty for Iceberg.

The Paimon catalog already handles the analogous case (Paimon primary keys ↔
Gravitino `PRIMARY_KEY` index) in both directions — see
`PaimonTable#constructIndexesFromPrimaryKeys` and
`GravitinoToPaimonTableConverter`. The Iceberg catalog should align with that
behavior.

### How should we improve?

Wire the Gravitino `PRIMARY_KEY` index through to / from Iceberg's
`identifier-field-ids`, forming a create/load round-trip closure:

- **Forward (create):** in `ConvertUtil#toIcebergSchema`, after building the
Iceberg `Schema`, set `identifier-field-ids` from the field ids of the
columns referenced by the single `PRIMARY_KEY` index, mirroring
`FlinkSchemaUtil.freshIdentifierFieldIds`. Iceberg requires identifier
fields to be `required` (NOT NULL); rely on Iceberg's own `Schema`
constructor to enforce that constraint (do not silently promote
nullability).
- **Reverse (load):** in `IcebergTable#fromIcebergTable`, reconstruct a
`PRIMARY_KEY` index from `Schema#identifierFieldIds()`, ordered by the
column position in the schema (Iceberg identifier fields are an unordered
set, so an explicit ordering is needed for determinism). Use a fixed
synthetic index name `ICEBERG_PRIMARY_KEY_INDEX`, analogous to
Paimon's `PAIMON_PRIMARY_KEY_INDEX`.
- **Validation:** replace the blanket "indexes not supported" rejection in
`IcebergCatalogOperations#createTable` with a focused check that allows at
most one `PRIMARY_KEY` index, with at least one column, and rejects nested
column references (`fieldName.length == 1`). All other `Index` shapes
(multiple indexes, non-`PRIMARY_KEY` types, nested columns) continue to be
rejected with a clear message.

Out of scope for this issue:

- Altering the primary key via `TableChange` — Iceberg's
`UpdateSchema#setIdentifierFields` exists, but PK evolution has additional
semantics (NOT NULL promotion of existing columns, interaction with
equality deletes) that warrant a separate discussion.
- Changes to the Iceberg REST server module (`iceberg-rest-server`).

I have a working patch with unit tests covering forward mapping, reverse
mapping, full round-trip, and validation rejection cases (multiple indexes,
non-PK index type, nested PK column, empty PK column list). Happy to open a
PR linked to this issue once it is accepted.

Contributor guide

Open the contributing guide

Research direction

Start with IcebergCatalogOperations#createTable, ConvertUtil#toIcebergSchema, and IcebergTable#fromIcebergTable, then compare PaimonTable#constructIndexesFromPrimaryKeys and GravitinoToPaimonTableConverter. Run the Iceberg catalog unit tests covering forward mapping, reverse mapping, round trips, and validation rejection cases. Done means PRIMARY_KEY metadata maps in both directions while unsupported index shapes remain rejected.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.