mars-project / mars-project/mars

[PROPOSAL] Mars storage lib

Open
#1,905 0 comments 0 reactions 0 assignees View on GitHub
mod: storage proposal
Dominant language
Python
Stars
2.7k
Forks
325
PR merge metrics
No merged PRs in 30d

Description

Currently, Mars storage needs to handle both memory and disk, we can define a abstract base class `StorageBackend` and implements backends like plasma, filesystem, vineyard, etc.

### API

StorageBackend defines as below:

``` Python
class StorageBackend(ABC):
@classmethod
@abstractmethod
async def setup(cls, **kwargs) -> Tuple[Dict, Dict]:
"""
Setup environments, for example, start plasma store for plasma backend.
Parameters
----------
kwargs : kwargs
Kwargs for setup.
Returns
-------
Tuple of two dicts
Dicts for initialization and teardown.
"""

@staticmethod
async def teardown(**kwargs):
"""
Clean up the environments.
Parameters
----------
kwargs : kwargs
Parameters for clean up.
"""

@property
@abstractmethod
def level(self):
"""
Level of current storage backend.
Returns
-------
Level: str
storage level.
"""

@abstractmethod
async def get(self, object_id, **kwargs) -> object:
"""
Get object by key. For some backends, `columns` or `slice` can pass to get part of data.
Parameters
----------
object_id : object id
Object id to get.
kwargs:
Additional keyword arguments
Returns
-------
Python object
"""

@abstractmethod
async def put(self, obj, importance=0) -> ObjectInfo:
"""
Put object into storage with object_id.
Parameters
----------
obj : python object
Object to put.
importance: int
The priority to spill when storage is full
Returns
-------
ObjectInfo
object information including size, raw_size, device
"""

@abstractmethod
async def delete(self, object_id):
"""
Delete object from storage by object_id.
Parameters
----------
object_id
object id
"""

@abstractmethod
async def object_info(self, object_id) -> ObjectInfo:
"""
Get information about stored object.
Parameters
----------
object_id
object id
Returns
-------
ObjectInfo
Object info including size, device and etc.
"""

@abstractmethod
async def open_writer(self, size=None) -> StorageFileObject:
"""
Return a file-like object for writing.
Parameters
----------
size: int
Maximum size in bytes
Returns
-------
fileobj: StorageFileObject
"""

@abstractmethod
async def open_reader(self, object_id) -> StorageFileObject:
"""
Return a file-like object for reading.
Parameters
----------
object_id
Object id
Returns
-------
fileobj: StorageFileObject
"""

async def list(self) -> List:
"""
List all stored objects in storage.
Returns
-------
List of objects
"""

async def prefetch(self, object_id):
"""
Fetch object to current worker.
Parameters
----------
object_id
Object id.
"""

async def pin(self, object_id):
"""
Pin the data to prevent the data being released or spilled.
Parameters
----------
object_id
object id
"""

async def unpin(self, object_id):
"""
Unpin the data, allow storage to release the data.
Parameters
----------
object_id
object id
"""
```

the definition of `StorageLevel`:

``` Python
class StorageLevel(Enum):
GPU = 1 << 0
MEMORY = 1 << 1
DISK = 1 << 2
REMOTE = 1 << 3

def __and__(self, other: "StorageLevel"):
return self.value | other.value
```

if the storage could handle both memory and disk, the level can be expressed as `StorageLevel.MEMORY & StorageLevel.DISK`.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing the existing Mars storage implementation, then compare it with the proposed StorageBackend and StorageLevel APIs; completion would require an agreed design and implementation plan for memory, disk, and other backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.