apache / apache/gravitino

[FEATURE] Supports iceberg metadata migration through the iceberg rest server

Open
#5,574 6 comments 1 reaction 0 assignees View on GitHub
feature
Dominant language
Java
Stars
3.2k
Forks
935
Avg merge
1d 16h
Merged PRs (30d)
298

Description

### Describe the feature

In our real-world scenario, a generic iceberg catalog name such as `iceberg_{cluster_name}` is used in a region, regardless of the specific iceberg backend metastore.

Users do not need to care about the underlying metadata storage service.

When we do the migration of the metadata service at the bottom of iceberg, users should not be aware of it, and they should not be asked to modify the iceberg catalog name in their sql, otherwise it will be very difficult to implement.

Therefore, we want to implicitly implement the switching of iceberg's underlying metadata storage service without requiring users to modify their sql in iceberg rest server.

**My initial thought is:**
supporting a `master-slave` backend catalog in the iceberg rest server,as shown in the figure.

![image](https://github.com/user-attachments/assets/a1a9947b-3da4-4716-970d-7d5cd21229ff)

1. For new tables, access master metastore directly to create, read, or modify metadata

2. For the old table, if it exists in master metastore, the metadata is read or modified directly based on master metastore.
If it does not exist in master metastore, copy the metadata from slave metastore to master metastore at first, and then read or modify the metadata based on master metastore.

### Motivation

_No response_

### Describe the solution

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by locating the Iceberg REST server entry point and its backend catalog handling; the issue names no files or tests. Clarify the master/slave migration behavior for new and existing tables, including how metadata is copied, and define acceptance criteria showing that users can keep the same SQL catalog name.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.