apache / apache/gravitino

[Improvement] Improve iceberg rest catalog service

Open
#6,034 2 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### What would you like to be improved?

The Iceberg REST Catalog Service is pivotal for our company's transition to a unified metadata system, and I believe it should be our primary focus (Only when the REST Catalog offers significantly more benefits compared to the Hive Catalog will we have the motivation to proceed with the metadata migration).
Here are some of my thoughts on potential improvements to the Iceberg REST Catalog Service:

1. Fine-grained lock orchestration on the service side: By orchestrating commits, we can reduce the number of lock calls and decrease the commit failure rate. Furthermore, for append-only semantics, it is actually possible to operate without locks.
2. Unified caching: From the service's perspective, we should add global caching for Iceberg tables.
3. Taking over more Iceberg metadata structures into the REST Catalog Service: this would not only provide higher access performance but also allow for unified handling of Iceberg version compatibility.

As we are currently evaluating options for a unified metadata solution, we are eager to actively participate in its development :)

### How should we improve?

Give me a moment to think

Contributor guide

Open the contributing guide

Research direction

The issue proposes fine-grained commit locking, global table caching, and broader Iceberg metadata handling, but names no files, tests, or entry points. First clarify which improvement is in scope and the REST-versus-Hive requirements; the issue does not yet define a concrete completion condition.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.