meta-pytorch / meta-pytorch/data

torchdata not compatible with torch 2.3.0

Open
#1,244 3 comments 21 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.3k
Forks
179
Avg merge
6d 1h
Merged PRs (30d)
2

Description

🐛 Describe the bug

Describe the bug

torchdata does not work with torch 2.3.0 because DILL_AVAILABLE is not available where expected:

Python 3.12.2 (main, Feb  6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchdata
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/__init__.py", line 9, in <module>
    from . import datapipes
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/__init__.py", line 9, in <module>
    from . import iter, map, utils
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/iter/__init__.py", line 79, in <module>
    from torchdata.datapipes.iter.util.cacheholder import (
  File "/private/tmp/test/lib/python3.12/site-packages/torchdata/datapipes/iter/util/cacheholder.py", line 24, in <module>
    from torch.utils.data.datapipes.utils.common import _check_unpickable_fn, DILL_AVAILABLE
ImportError: cannot import name 'DILL_AVAILABLE' from 'torch.utils.data.datapipes.utils.common' (/private/tmp/test/lib/python3.12/site-packages/torch/utils/data/datapipes/utils/common.py)
>>>

This was fixed in https://github.com/pytorch/pytorch/pull/122616 but the fix did not make it into the torch 2.3.0 release
https://github.com/pytorch/pytorch/blob/v2.3.0/torch/utils/data/datapipes/utils/common.py

I would assume 2.3.1 will be ok because the fix is in main. For now though, a clean install of torchdata is broken.

I would recommend adding a torch>=2,!=2.3.0 to setup.py

To Reproduce

pip install torchdata
import torchdata

Expected behavior

successful import

Environment

fresh install

Versions

Collecting environment information...
PyTorch version: 2.3.0
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: macOS 14.4.1 (arm64)
GCC version: Could not collect
Clang version: 15.0.0 (clang-1500.3.9.4)
CMake version: version 3.27.1
Libc version: N/A

Python version: 3.12.2 (main, Feb 6 2024, 20:19:44) [Clang 15.0.0 (clang-1500.1.0.2.5)] (64-bit runtime)
Python platform: macOS-14.4.1-arm64-arm-64bit
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Apple M1 Max

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] torch==2.3.0
[pip3] torchdata==0.7.1
[conda] Could not collect

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with setup.py and the torchdata import path shown in the traceback, then reproduce the failure with torch 2.3.0 using pip install torchdata and import torchdata. Update the dependency constraints to exclude the incompatible torch release, and verify that installation and import succeed with supported versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
data, machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.