meta-pytorch / meta-pytorch/data
Add memmap cache for Tensor
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🚀 The feature
Beyond the on-disk cache and in-memory cache, it would be useful and performant if a memmap cache (under tensordict https://github.com/pytorch-labs/tensordict/blob/main/tensordict/memmap.py)
It would boost better performance due to
- Reduce the overhead of the inter-process communication
- Reduce the time of decoding, etc. Users would directly read Tensor after the first epoch.
However, there are two major limitations:
- Input has to be Tensor
- The whole dataset has to fit into local fs
Motivation, pitch
Performance
Alternatives
No response
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading tensordict/memmap.py to understand the referenced memmap implementation and compare it with this repository's existing on-disk and in-memory caches. Done should provide a Tensor-only memmap cache that reduces inter-process communication and decoding overhead while documenting or enforcing the local-filesystem size limitation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100