apache / apache/gravitino

[EPIC] Idempotency-Key support for Iceberg REST server (multi-node safe)

Open
#12,049 0 comments 0 reactions 0 assignees View on GitHub
epic
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 17h
Merged PRs (30d)
339

Description

## Describe the proposal

The Iceberg REST spec defines an optional `Idempotency-Key` header (UUIDv7) on mutation endpoints. When present, the server must return the original response for replayed keys instead of re-executing the operation, preventing duplicate side-effects from client retries.

This epic covers the full implementation of idempotency support in Gravitino's Iceberg REST server, including multi-node consistency and node-failure resilience.

Key concerns addressed:
- **Multi-node deployments**: Retries may hit different nodes behind a load balancer; a shared durable store (JDBC-backed) ensures all nodes see the same idempotency state
- **Node failure resilience**: DB-backed records survive crashes; TTL-based expiry handles abandoned reservations
- **Configurable storage backends**: In-memory (single-node/dev) and JDBC-backed (production)
- **Spec compliance**: UUIDv7 validation, `idempotency-key-lifetime` advertised in `GET /v1/config`

Reference: [apache/polaris#3205](https://github.com/apache/polaris/pull/3205) (database-backed IdempotencyStore)

Design document: `design-docs/iceberg-idempotency-key.md`

Supersedes #10683.

## Task list

- [ ] Design document review and approval
- [ ] IdempotencyStore SPI interface, IdempotencyRecord model, and InMemoryIdempotencyStore
- [ ] JdbcIdempotencyStore using Gravitino's relational backend (H2/MySQL/PostgreSQL)
- [ ] Header handling, replay logic, and mutation endpoint integration
- [ ] Advertise `idempotency-key-lifetime` in `GET /v1/config`
- [ ] Unit and integration tests

Contributor guide

Open the contributing guide

Research direction

Start with design-docs/iceberg-idempotency-key.md and review the task list for the proposed IdempotencyStore SPI, record model, storage backends, endpoint integration, configuration, and tests. Done means the design is approved and all listed multi-node, failure-resilience, spec-compliance, and unit/integration-test work is complete.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql, postgresql
Domain
api, backend, databases, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.