meta-pytorch / meta-pytorch/data

Utilize `multiprocessing.shared_memory` in `DataLoader2` for Performance Improvements

Open
#1,031 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🚀 The feature

Given that we will not support for Python 3.7 in future releases, we can utilize multiprocessing.shared_memory that was introduced in Python 3.8.

It can potentially replaces some of the existing interprocess communication that we have, notably the usage of serialization/deserialization and queues. We will first have to evaluate the feasibility; it is not clear to me how flexible it is (i.e. if it can store arbitrary objects without serialization).

Some possible areas of improvements include:

  1. Worker prefetchers can write to shared memory that can be accessed by main process, rather than its own buffer, then serialize/deserialize, and put into a queue
  2. Single dispatch mechanism can write to shared memory that is readable by worker processes, skipping over most of the existing IPC
Motivation, pitch

If used correctly, it has the potential of greatly improving multiprocessing data loading performance.

Alternatives

Keep things as they are if we do not find improvements or if it is too complex/premature to use.

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the DataLoader2 worker prefetchers and single dispatch mechanism, then review how their current serialization, buffers, and queues provide interprocess communication. Evaluate whether multiprocessing.shared_memory can support the required data and measure its effect on multiprocessing data-loading performance. Done means a feasibility result with evidence for adopting or retaining the current approach.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.