meta-pytorch / meta-pytorch/data
Unable to vectorize datapipe operations
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
Let t be an input dataset that associates strings (model input) to integers (model output):
t = [("a", 567), ("b", 908), ("c", 887)]
I now wrap t in a SequenceWrapper, to use it as part of a DataPipe:
import torchdata.datapipes as dp
pipeline = dp.map.SequenceWrapper(t, deepcopy=False)
Now, I have a datapipe giving me tuples:
>>> pipeline[0]
('a', 567)
After that, I am willing to do some preprocessing. However, since I have a huge dataset I want to vectorize the following operations: for that, I use .batch:
batched_pipeline = pipeline.batch(batch_size=2)
By vectorizing, I mean grouping the X values (the strings) and the Y values (integers) together so that I can apply a custom logic to the input and the output at the same time, and in batch.
However, the .batch() function returns the following:
>>> batched_pipeline[0]
[('a', 567), ('b', 908)]
Which really makes no sense because why would I want the whole line batched? Just so that I can iterate over it right after?
In my opinion, .batch() only makes sense if the different slices (see TensorFlow's Dataset.from_tensor_slices() which does handle that) are batched separately.
So what do you think? Is there something I am missing?
Thanks in advance!
Versions
PyTorch version: 1.12.0+cu116
Is debug build: False
CUDA used to build PyTorch: 11.6
ROCM used to build PyTorch: N/A
OS: Red Hat Enterprise Linux release 8.5 (Ootpa) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-3)
Clang version: 12.0.1 (Red Hat 12.0.1-2.module+el8.5.0+12651+6a7729ff)
CMake version: version 3.20.2
Libc version: glibc-2.28
Python version: 3.10.4 (main, Mar 31 2022, 08:41:55) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-4.18.0-348.el8.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB
Nvidia driver version: 515.48.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] light-the-torch==0.4.0
[pip3] mypy-extensions==0.4.3
[pip3] numpy==1.23.0
[pip3] torch==1.12.0+cu116
[pip3] torchaudio==0.12.0
[pip3] torchdata==0.4.0
[pip3] torchmetrics==0.9.2
[pip3] torchtext==0.13.0
[pip3] torchvision==0.13.0
[conda] light-the-torch 0.4.0 pypi_0 pypi
[conda] numpy 1.23.0 pypi_0 pypi
[conda] torch 1.12.0+cu116 pypi_0 pypi
[conda] torchaudio 0.12.0 pypi_0 pypi
[conda] torchdata 0.4.0 pypi_0 pypi
[conda] torchmetrics 0.9.2 pypi_0 pypi
[conda] torchtext 0.13.0 pypi_0 pypi
[conda] torchvision 0.13.0 pypi_0 pypi
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the SequenceWrapper and batch entry points shown in the report, then run the minimal Python example to trace the produced batch structure. Check the existing DataPipe behavior and documentation to determine whether tuple fields should be grouped or whether the reported result is expected; done means the intended behavior is clarified and covered by the relevant change or documentation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100