meta-pytorch / meta-pytorch/data

MapDatapipe Mux/Demux Support

Open
#310 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🚀 The feature

MapDatapipes are missing Mux and Demux pipes as noted in https://github.com/pytorch/pytorch/issues/57031

Talked to @ejguan on https://discuss.pytorch.org/t/mapdatapipe-support-mux-demux/146305, I plan to do a PR with Mux/Demux added. However, I will add rough outlines / ideas here first. I plan to match the same test strategy as the Mux/Demux pipes already in IterDataPipes.

Motivation, pitch

For Demux: My basic test/goal is to download mnist, and split it into train/validation sets using map.
For Mux: Then attempt to mux them back together (not sure how to come up with a useful example of this).
- Might try a scenario where I split train into k splits and rejoin them?

Not sure when this should be converted to a pr. This would be my first pr into pytorch, so I want the pr to be as clean as possible. Putting code changes ideas here I feel could allow for more dramatic/messy changes/avoid a messy git diff/worry about formatting once code is finalized.

Note: doc strings are removed to make code shorter and will be readded in pr. Not-super-useful comments will be removed in pr.

Note: let me know if a draft pr would be better.

Demux working code:
Draft 1: https://github.com/josiahls/fastrl/blob/848f90d0ed5b0c2cd0dd3e134b0b922dd8a53d7c/fastrl/fastai/data/pipes.py

Demux working code + Basic Test
Draft 1: https://github.com/josiahls/fastrl/blob/848f90d0ed5b0c2cd0dd3e134b0b922dd8a53d7c/nbs/02c_fastai.data.pipes.ipynb

Mux working code:
Draft 1: https://github.com/josiahls/fastrl/blob/30cd47766e9fb1bc75d32de877f54b8de9567c36/fastrl/fastai/data/pipes/mux.py

Basic Test
Draft 1: https://github.com/josiahls/fastrl/blob/30cd47766e9fb1bc75d32de877f54b8de9567c36/nbs/02c_fastai.data.pipes.mux.ipynb

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing the existing Mux/Demux pipes and their test strategy in IterDataPipes with the MapDataPipes design described here, then review the linked Demux and Mux drafts. Done means adding MapDataPipe Mux and Demux support with matching tests, including the proposed MNIST split and recombination scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.