[Improvement] Improve iceberg rest catalog service
- Dominant language
- Java
- Stars
- 3.2k
- Forks
- 935
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 298
Description
### What would you like to be improved?
The Iceberg REST Catalog Service is pivotal for our company's transition to a unified metadata system, and I believe it should be our primary focus (Only when the REST Catalog offers significantly more benefits compared to the Hive Catalog will we have the motivation to proceed with the metadata migration).
Here are some of my thoughts on potential improvements to the Iceberg REST Catalog Service:
1. Fine-grained lock orchestration on the service side: By orchestrating commits, we can reduce the number of lock calls and decrease the commit failure rate. Furthermore, for append-only semantics, it is actually possible to operate without locks.
2. Unified caching: From the service's perspective, we should add global caching for Iceberg tables.
3. Taking over more Iceberg metadata structures into the REST Catalog Service: this would not only provide higher access performance but also allow for unified handling of Iceberg version compatibility.
As we are currently evaluating options for a unified metadata solution, we are eager to actively participate in its development :)
### How should we improve?
Give me a moment to think
Contributor guide
Research direction
The issue proposes fine-grained commit locking, global table caching, and broader Iceberg metadata handling, but names no files, tests, or entry points. First clarify which improvement is in scope and the REST-versus-Hive requirements; the issue does not yet define a concrete completion condition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100