[FEATURE] Doris catalog: support Key Model specification (PRIMARY KEY / UNIQUE KEY / AGGREGATE KEY / DUPLICATE KEY)
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### Describe the feature
The Doris catalog does not support Doris table Key Models (`PRIMARY KEY`, `UNIQUE KEY`, `AGGREGATE KEY`, `DUPLICATE KEY`).
**On `loadTable`**: All four Key Models are silently lost. The table's indexes and sort orders do not reflect the key columns. There is no warning or error.
**On `createTable`**: Gravitino never emits a Key Model clause, so Doris defaults to `DUPLICATE KEY()`. A user who loads a UNIQUE KEY table and recreates it through Gravitino will get a DUPLICATE KEY table — losing all uniqueness guarantees.
### Motivation
Key Model is a core semantic property of a Doris table. It determines:
- Whether duplicate rows are allowed (DUPLICATE) or rejected (UNIQUE / PRIMARY).
- How data is pre-aggregated on write (AGGREGATE KEY with SUM / REPLACE / HLL functions).
- Which columns define the table's sort order for data layout.
Without Key Model support, tables created through Gravitino lose their intended data integrity semantics. Applications that rely on specific Key Models for correctness cannot use Gravitino as the table creation path.
### Describe the solution
The Key Model information is present in `SHOW CREATE TABLE` output (e.g., `UNIQUE KEY(id, name)`) but is currently not parsed. The table-level Key Model is not available through existing index metadata, which only covers secondary indexes.
The fix needs to:
1. Parse the Key Model clause from the table DDL in the load path.
2. Generate the Key Model clause in the createTable DDL generation path.
3. Map Key Model to Gravitino's table model — this may need API-level discussion since Key Model isn't a concept shared across all JDBC catalogs.
The aggregate function types for AGGREGATE KEY columns (SUM, REPLACE, HLL, BITMAP_UNION) also need to be preserved during round-trip.
### Test plan
| # | Scenario | Expected after fix |
|---|----------|-------------------|
| 1 | DUPLICATE KEY table → loadTable | indexes reflect DUPLICATE KEY columns |
| 2 | UNIQUE KEY (single column) → loadTable | indexes reflect UNIQUE KEY, sort orders correct |
| 3 | UNIQUE KEY (multi-column) → loadTable | all key columns present in indexes |
| 4 | AGGREGATE KEY + SUM function → loadTable | KEY columns + aggregate function type preserved |
| 5 | createTable via Gravitino with KEY specification | DDL contains correct KEY clause |
| 6 | Round-trip: create → load → KEY matches original | no semantic loss |
| 7 | Table without explicit KEY (default) | no regression, behavior unchanged |
### Additional context
**Test results** (Docker + Gravitino 1.2.0, before fix):
| # | Scenario | Gravitino loadTable | Native SHOW CREATE TABLE |
|---|----------|:-------------------:|:------------------------:|
| 1 | DUPLICATE KEY(`id`) | ❌ `indexes: []` | ✅ `DUPLICATE KEY(\`id\`)` |
| 2 | UNIQUE KEY(`id`) | ❌ `indexes: []` | ✅ `UNIQUE KEY(\`id\`)` |
| 3 | UNIQUE KEY(`id`, `name`) | ❌ `indexes: []` | ✅ `UNIQUE KEY(\`id\`, \`name\`)` |
| 4 | AGGREGATE KEY(`site`, `city`) + `pv BIGINT SUM` | ❌ `indexes: []`, SUM lost | ✅ `AGGREGATE KEY(...)`, `pv bigint SUM` |
| 5 | Gravitino createTable | ❌ no KEY → defaults to `DUPLICATE KEY(all cols)` | — |
All four Key Models return empty indexes across all tested Doris versions. Multi-column keys and aggregate column functions are also lost.
I'd like to pick this up. Please let me know if you have any feedback.
Contributor guide
Research direction
Start at the Doris catalog's loadTable and createTable DDL-generation paths, then inspect SHOW CREATE TABLE parsing and the table-model/API mapping discussed in the issue. Validate against the seven scenarios in the test plan, including key columns, sort orders, aggregate functions, and no-key behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100