Distributed Key Value Tensor Store
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 206
- PR merge metrics
- No merged PRs in 30d
Description
Is it possible to use streaming dataset as a distributed key value store?
i have a set of keys (strings like "xyz_123") each that correspond to an numpy array
ideally I can do something like
np_array = dataset["xyz_123"]
but i see with MDSWriter.write that the keys of the dataset are just sequential and i can't change them.
Is there a way to have a custom key for MDSWriter?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the streaming dataset access path and MDSWriter, focusing on how sequential keys are assigned and looked up. Compare that behavior with the requested dataset["xyz_123"] access for NumPy arrays. Done means the issue's custom-key use case is supported or its limitation is clearly documented, with relevant tests added.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- numpy, python
- Domain
- data, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100