Imageomics / Imageomics/hpc-inference
`ImageFolderDataset` is not picklable: `DataLoader(num_workers>0)` fails under the `spawn` start method (macOS/Windows)
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
# Terms
- Pickle / picklable: Python's built-in serialization (`pickle`). An object is *picklable* if it can be converted to a byte stream and rebuilt elsewhere. Functions are pickled by qualified name, not by their code, so the target must be importable by that name.
- lambda: an anonymous function. One defined inside another function lives in that function's local scope, which has no importable name. Lambda functions are therefore not picklable.
- Bound method: a method accessed on an instance such as `self.foo`. It's picklable: pickle stores the method name plus the instance, and rebuilds it on the other side.
- Multiprocessing start method: how Python launches a child process:
- `fork` Linux default: the child is a copy of the parent's memory; **objects are inherited, not pickled.**
- `spawn` macOS & Windows default: the child starts fresh and the parent pickles the objects it needs to send over. Anything unpicklable fails here.
- `DataLoader` workers: when `num_workers > 0`, PyTorch runs data loading in child process. Under `spawn`, it must pickle the `Dataset` to hand it to each worker.
# Summary
`ImageFolderDataset` can't be sent to `DataLoader` worker processes on platforms where multiprocessing uses the `spawn` start method (macOS, Windows). Any use with `num_workers > 0` raises:
```py
AttributeError: Can't get local object
'ImageFolderDataset._init_uuid_generator..'
```
It works on Linux only because the default start method there is fork (workers inherit memory and the dataset is never pickled). That's why it wasn't an issue for us during development on OSC computers.
# Root Cause
[`_init_uuid_generator`](https://github.com/Imageomics/hpc-inference/blob/e0236778eca974123eebad84168b2bb143f7dddc/src/hpc_inference/datasets/image_folder_dataset.py#L68-L83) assigns the UUID function to a lambda defined in a local scope:
```py
def _init_uuid_generator(self, uuid_mode):
if uuid_mode == "filename":
self.generate_uuid = lambda img_path: os.path.basename(img_path)
elif uuid_mode == "relative":
parent_dir = os.path.dirname(self.image_dir)
self.generate_uuid = lambda img_path: os.path.relpath(img_path, parent_dir)
elif uuid_mode == "fullpath":
self.generate_uuid = lambda img_path: img_path # <-- unpicklable
...
```
Reproduction
```py
import pickle
from hpc_inference.datasets.image_folder_dataset import ImageFolderDataset
ds = ImageFolderDataset(image_dir="some_folder", uuid_mode="fullpath")
pickle.dumps(ds)
# AttributeError: Can't get local object
# 'ImageFolderDataset._init_uuid_generator..'
```
Equivalently, on macOS:
```py
import torch
loader = torch.utils.data.DataLoader(ds, num_workers=4)
for _ in loader: # raises at worker startup
pass
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.