mars-project / mars-project/mars
[storage] Add batch.put API for storage
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.7k
- Forks
- 325
- PR merge metrics
- No merged PRs in 30d
Description
**Is your feature request related to a problem? Please describe.**
`Ray.put` with owner will issue an sync rpc to owner, when there are many obejcts to put, `Ray.put` will become the bottleneck for operand executing especially for shuffle mapper stage.
**Describe the solution you'd like**
A `batch.put` API will eliminate the issue.
With batch put:
without batch put
The api can be added to `mars.storage.base.StorageBackend`:
```
@abstractmethod
async def batch_put(self, objects, importance: int = 0) -> List[ObjectInfo]:
return [self.put(obj, importance) for obj in objects]
```
If the storage client supports batch.put, it can override this method to get better performance.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading mars.storage.base.StorageBackend and its existing put implementations. Trace how storage clients handle multiple objects and identify which clients can support a batch.put operation. Done means the batch_put API is defined and supported clients can use it without the repeated synchronous owner RPC bottleneck.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100