mars-project / mars-project/mars

Refactor of storage service

Open
#2,352 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

mod: storage type: enhancement
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

This proposal aims to improve the stability of storage service and clarify the responsibilities of each actors created by storage service.

#### Actors on main pool

- StorageManagerActor
- Create all necessary actors for both main pool and subpools.
- Preserve setup params and teardown params for all storage backends.
- Report storage status to cluster API
- main APIs
- `async def upload_storage_info(self):`
- `async def upload_disk_info(self):`
- RefHolderActor
- Hold the references for stored objects to prevent being evicted by storage backends like `plasma`.
- mian APIs
- `async def ref_object(session_id: str, data_key: str, level: StorageLevel, object_id: ObjectID):`
- `async def release_object(session_id: str, data_key: str, level: StorageLevel):`
- StorageQuotaActor
- Manage the quotas of all storage backends.
- main APIs
- `def request_quota(self, size: int) -> bool:`
- `def release_quota(self, size: int):`
- `def update_quota(self, size: int):`
- `def get_quota_info(self) -> Tuple[float, float]:`
- SpillManagerActor
- Manager spill tasks and decides which keys to be spilled. It records the keys who are spilling or spilled.
- API
- `def get_spill_keys(request_size: int, level: StorageLevel, band_name: str, multiplier=1.1)`
- TransferManagerActor
- Manage transfer tasks and records the information of transfer tasks.
- API
- `def get_fetch_infos(session_id: str, data_keys: List[str])`

#### Actors on subpools

- StorageHandlerActor
- The handler actor for each subpool. It holds clients of all storage backends and provides methods like `get`, `put`, etc.
- It keeps the same APIs as older version.
- SpillActor
- Called by `StorageHandlerActor` when spilling is triggered, normally created on IO processes, also necessary for GPU band processes.
- main APIs
- `def spill(self, session_id: str, data_keys: List[int], level: StorageLevel, block_size=None):`
- SenderActor
- Called by `StorageHandlerActor`, normally created on IO processes, also necessary for GPU band processes.
- It keeps the same APIs as older version.
- ReceiverActor
- Called by `SenderActor`, normally created on IO processes, also necessary for GPU band processes.
- It keeps the same APIs as older version.

#### Other changes

- Remove `DataManagerActor`, all storage meta will be stored in meta service on each worker, use `MetaAPI` to delete and update storage meta.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the existing storage service and DataManagerActor, then map the responsibilities and APIs listed for the main-pool and subpool actors. Done would require an agreed actor design, removal of DataManagerActor, migration of storage metadata to the worker meta service through MetaAPI, and corresponding validation; no files or tests are named.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.