apache / apache/gravitino

[FEATURE] Doris catalog: support Key Model specification (PRIMARY KEY / UNIQUE KEY / AGGREGATE KEY / DUPLICATE KEY)

Open
#11,975 0 comments 1 reaction 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the feature

The Doris catalog does not support Doris table Key Models (`PRIMARY KEY`, `UNIQUE KEY`, `AGGREGATE KEY`, `DUPLICATE KEY`).

**On `loadTable`**: All four Key Models are silently lost. The table's indexes and sort orders do not reflect the key columns. There is no warning or error.

**On `createTable`**: Gravitino never emits a Key Model clause, so Doris defaults to `DUPLICATE KEY()`. A user who loads a UNIQUE KEY table and recreates it through Gravitino will get a DUPLICATE KEY table — losing all uniqueness guarantees.

### Motivation

Key Model is a core semantic property of a Doris table. It determines:
- Whether duplicate rows are allowed (DUPLICATE) or rejected (UNIQUE / PRIMARY).
- How data is pre-aggregated on write (AGGREGATE KEY with SUM / REPLACE / HLL functions).
- Which columns define the table's sort order for data layout.

Without Key Model support, tables created through Gravitino lose their intended data integrity semantics. Applications that rely on specific Key Models for correctness cannot use Gravitino as the table creation path.

### Describe the solution

The Key Model information is present in `SHOW CREATE TABLE` output (e.g., `UNIQUE KEY(id, name)`) but is currently not parsed. The table-level Key Model is not available through existing index metadata, which only covers secondary indexes.

The fix needs to:
1. Parse the Key Model clause from the table DDL in the load path.
2. Generate the Key Model clause in the createTable DDL generation path.
3. Map Key Model to Gravitino's table model — this may need API-level discussion since Key Model isn't a concept shared across all JDBC catalogs.

The aggregate function types for AGGREGATE KEY columns (SUM, REPLACE, HLL, BITMAP_UNION) also need to be preserved during round-trip.

### Test plan

| # | Scenario | Expected after fix |
|---|----------|-------------------|
| 1 | DUPLICATE KEY table → loadTable | indexes reflect DUPLICATE KEY columns |
| 2 | UNIQUE KEY (single column) → loadTable | indexes reflect UNIQUE KEY, sort orders correct |
| 3 | UNIQUE KEY (multi-column) → loadTable | all key columns present in indexes |
| 4 | AGGREGATE KEY + SUM function → loadTable | KEY columns + aggregate function type preserved |
| 5 | createTable via Gravitino with KEY specification | DDL contains correct KEY clause |
| 6 | Round-trip: create → load → KEY matches original | no semantic loss |
| 7 | Table without explicit KEY (default) | no regression, behavior unchanged |

### Additional context

**Test results** (Docker + Gravitino 1.2.0, before fix):

| # | Scenario | Gravitino loadTable | Native SHOW CREATE TABLE |
|---|----------|:-------------------:|:------------------------:|
| 1 | DUPLICATE KEY(`id`) | ❌ `indexes: []` | ✅ `DUPLICATE KEY(\`id\`)` |
| 2 | UNIQUE KEY(`id`) | ❌ `indexes: []` | ✅ `UNIQUE KEY(\`id\`)` |
| 3 | UNIQUE KEY(`id`, `name`) | ❌ `indexes: []` | ✅ `UNIQUE KEY(\`id\`, \`name\`)` |
| 4 | AGGREGATE KEY(`site`, `city`) + `pv BIGINT SUM` | ❌ `indexes: []`, SUM lost | ✅ `AGGREGATE KEY(...)`, `pv bigint SUM` |
| 5 | Gravitino createTable | ❌ no KEY → defaults to `DUPLICATE KEY(all cols)` | — |

All four Key Models return empty indexes across all tested Doris versions. Multi-column keys and aggregate column functions are also lost.

I'd like to pick this up. Please let me know if you have any feedback.

Contributor guide

Open the contributing guide

Research direction

Start at the Doris catalog's loadTable and createTable DDL-generation paths, then inspect SHOW CREATE TABLE parsing and the table-model/API mapping discussed in the issue. Validate against the seven scenarios in the test plan, including key columns, sort orders, aggregate functions, and no-key behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.