meta-pytorch / meta-pytorch/data
Improve the warning message regarding local function not supported by pickle
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.3k
- Forks
- 179
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 2
Description
🐛 Describe the bug
I used a local function in a pipeline which results in the warning:
UserWarning: Local function is not supported by pickle, please use regular python function or functools.partial instead.
So I wrapped the local function in functools.partial and the warning persisted, so I investigated closer. The problem appears to be that _check_unpickable_fn is undoing the partial before checking if fn is local, i.e.
This code unwraps partial, converting back to the underlying local function
And this code then produces the warning
My calling code is
def run(args)
def batch_transform(x):
return {"inputs": text_transform(x["text"]), "target": label_transform(x["label"])}
train_datapipe,, test_datapipe= DATASETS[args.dataset](root=args.data_dir, split=('train', 'test'))
# shuffle and batch
train_datapipe = train_datapipe.shuffle().batch(args.batch_size).rows2columnar(["text", "label"])
train_datapipe = train_datapipe.map(partial(batch_transform))
This produces the warning regardless of whether batch_transform is partial or not (a fix is to import dill or lift batch_transform out of run)
Versions
PyTorch version: 1.13.0+cu117
Is debug build: False
CUDA used to build PyTorch: 11.7
ROCM used to build PyTorch: N/A
OS: Ubuntu 22.04.1 LTS (x86_64)
GCC version: (Ubuntu 11.3.0-1ubuntu1~22.04) 11.3.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35
Python version: 3.10.6 (main, Nov 14 2022, 16:10:14) [GCC 11.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-58-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.99
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 2080 Ti
Nvidia driver version: 525.60.13
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] pytorch-ignite==0.4.10
[pip3] torch==1.13.0+cu117
[pip3] torchdata==0.5.0
[pip3] torchtext==0.14.0
[conda] Could not collect
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in torch/utils/data/datapipes/utils/common.py at _check_unpickable_fn and reproduce the warning with a local function wrapped in functools.partial. Review how partials are unwrapped before the local-function check; done means the warning behavior or wording accurately reflects the callable being passed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- data
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100