apache / apache/amoro

[Feature]: Support AMS horizontal performance expansion capabilities

Open
#3,670 1 comment 0 reactions 0 assignees View on GitHub
type:feature
Dominant language
Java
Stars
1.2k
Forks
395
Avg merge
4d 10h
Merged PRs (30d)
33

Description

### Description

Currently, AMS uses a master-slave mode. If a large number of Iceberg tables are managed, the master-slave mode will have performance bottlenecks, and a solution that can support horizontal performance expansion is needed.

### Use case/motivation

Multiple catalogs need to be registered on amoro. Under each catalog, there are many Iceberg tables that need to be managed automatically. If only a single-node AMS is working, the single node will consume a lot of resources. It is necessary to support a horizontally scalable performance expansion solution. By expanding the capacity of AMS nodes, each node can be responsible for processing a part of the Iceberg table. In this way, the resources of each node can be controlled and horizontal performance expansion can be achieved.

### Describe the solution

Extended support for a new master-slave mode. Each AMS node is only responsible for processing a part of the Iceberg table. It supports dynamic expansion and contraction of the number of AMS nodes and automatic migration of the Iceberg table. The optimizer will pull tasks from each AMS node for processing. The AMS master node is responsible for evenly distributing the Iceberg table to all AMS nodes in the cluster and managing the life cycle of the AMS nodes in the cluster.

### Subtasks

- [x] #3845
- [x] #3782
- [x] #3919
- [x] #3921
- [x] #4121
- [x] #3923
- [x] #3928
- [x] #3929
- [x] #4171
- [ ] In master-slave mode, for table migration scenarios, support committing routes to the new node.

### Related issues

_No response_

### Are you willing to submit a PR?

- [x] Yes I am willing to submit a PR!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

Open the contributing guide

Research direction

The issue names no source files, tests, or entry points; begin by locating AMS master-slave table assignment and route-commit handling. Done means migration scenarios can commit routes to the new node while the master distributes tables and cluster membership can expand or contract.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.