[Feature]: Support AMS horizontal performance expansion capabilities
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 395
- Avg merge
- 4d 10h
- Merged PRs (30d)
- 33
Description
### Description
Currently, AMS uses a master-slave mode. If a large number of Iceberg tables are managed, the master-slave mode will have performance bottlenecks, and a solution that can support horizontal performance expansion is needed.
### Use case/motivation
Multiple catalogs need to be registered on amoro. Under each catalog, there are many Iceberg tables that need to be managed automatically. If only a single-node AMS is working, the single node will consume a lot of resources. It is necessary to support a horizontally scalable performance expansion solution. By expanding the capacity of AMS nodes, each node can be responsible for processing a part of the Iceberg table. In this way, the resources of each node can be controlled and horizontal performance expansion can be achieved.
### Describe the solution
Extended support for a new master-slave mode. Each AMS node is only responsible for processing a part of the Iceberg table. It supports dynamic expansion and contraction of the number of AMS nodes and automatic migration of the Iceberg table. The optimizer will pull tasks from each AMS node for processing. The AMS master node is responsible for evenly distributing the Iceberg table to all AMS nodes in the cluster and managing the life cycle of the AMS nodes in the cluster.
### Subtasks
- [x] #3845
- [x] #3782
- [x] #3919
- [x] #3921
- [x] #4121
- [x] #3923
- [x] #3928
- [x] #3929
- [x] #4171
- [ ] In master-slave mode, for table migration scenarios, support committing routes to the new node.
### Related issues
_No response_
### Are you willing to submit a PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
Contributor guide
Research direction
The issue names no source files, tests, or entry points; begin by locating AMS master-slave table assignment and route-commit handling. Done means migration scenarios can commit routes to the new node while the master distributes tables and cluster membership can expand or contract.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100