mars-project / mars-project/mars
Refactor of storage service
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
This proposal aims to improve the stability of storage service and clarify the responsibilities of each actors created by storage service.
#### Actors on main pool
- StorageManagerActor
- Create all necessary actors for both main pool and subpools.
- Preserve setup params and teardown params for all storage backends.
- Report storage status to cluster API
- main APIs
- `async def upload_storage_info(self):`
- `async def upload_disk_info(self):`
- RefHolderActor
- Hold the references for stored objects to prevent being evicted by storage backends like `plasma`.
- mian APIs
- `async def ref_object(session_id: str, data_key: str, level: StorageLevel, object_id: ObjectID):`
- `async def release_object(session_id: str, data_key: str, level: StorageLevel):`
- StorageQuotaActor
- Manage the quotas of all storage backends.
- main APIs
- `def request_quota(self, size: int) -> bool:`
- `def release_quota(self, size: int):`
- `def update_quota(self, size: int):`
- `def get_quota_info(self) -> Tuple[float, float]:`
- SpillManagerActor
- Manager spill tasks and decides which keys to be spilled. It records the keys who are spilling or spilled.
- API
- `def get_spill_keys(request_size: int, level: StorageLevel, band_name: str, multiplier=1.1)`
- TransferManagerActor
- Manage transfer tasks and records the information of transfer tasks.
- API
- `def get_fetch_infos(session_id: str, data_keys: List[str])`
#### Actors on subpools
- StorageHandlerActor
- The handler actor for each subpool. It holds clients of all storage backends and provides methods like `get`, `put`, etc.
- It keeps the same APIs as older version.
- SpillActor
- Called by `StorageHandlerActor` when spilling is triggered, normally created on IO processes, also necessary for GPU band processes.
- main APIs
- `def spill(self, session_id: str, data_keys: List[int], level: StorageLevel, block_size=None):`
- SenderActor
- Called by `StorageHandlerActor`, normally created on IO processes, also necessary for GPU band processes.
- It keeps the same APIs as older version.
- ReceiverActor
- Called by `SenderActor`, normally created on IO processes, also necessary for GPU band processes.
- It keeps the same APIs as older version.
#### Other changes
- Remove `DataManagerActor`, all storage meta will be stored in meta service on each worker, use `MetaAPI` to delete and update storage meta.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the existing storage service and DataManagerActor, then map the responsibilities and APIs listed for the main-pool and subpool actors. Done would require an agreed actor design, removal of DataManagerActor, migration of storage metadata to the worker meta service through MetaAPI, and corresponding validation; no files or tests are named.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100