meta-pytorch / meta-pytorch/data
Utilize `multiprocessing.shared_memory` in `DataLoader2` for Performance Improvements
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🚀 The feature
Given that we will not support for Python 3.7 in future releases, we can utilize multiprocessing.shared_memory that was introduced in Python 3.8.
It can potentially replaces some of the existing interprocess communication that we have, notably the usage of serialization/deserialization and queues. We will first have to evaluate the feasibility; it is not clear to me how flexible it is (i.e. if it can store arbitrary objects without serialization).
Some possible areas of improvements include:
- Worker prefetchers can write to shared memory that can be accessed by main process, rather than its own buffer, then serialize/deserialize, and put into a queue
- Single dispatch mechanism can write to shared memory that is readable by worker processes, skipping over most of the existing IPC
Motivation, pitch
If used correctly, it has the potential of greatly improving multiprocessing data loading performance.
Alternatives
Keep things as they are if we do not find improvements or if it is too complex/premature to use.
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the DataLoader2 worker prefetchers and single dispatch mechanism, then review how their current serialization, buffers, and queues provide interprocess communication. Evaluate whether multiprocessing.shared_memory can support the required data and measure its effect on multiprocessing data-loading performance. Done means a feasibility result with evidence for adopting or retaining the current approach.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100