Project-MONAI / Project-MONAI/MONAI

Recursion Error when setting DataLoader Random Seed with Hydra configuration

Open
#8,087 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
8.7k
Forks
1.6k
Avg merge
5d 1h
Merged PRs (30d)
20

Description

Describe the bug
When creating a monai dataloader inside a hydra app, I get the following recursion error

RecursionError: maximum recursion depth exceeded in comparison
    full_key: set_random_state
    object_type=<DataclassName>

I have created a minimum working example below.

To Reproduce
Steps to reproduce the behavior:

  1. create the following script monai_hydra.py

from monai.data import DataLoader
from hydra.core.config_store import ConfigStore
from dataclasses import dataclass, field
from monai.data.dataset import  Dataset
import torch
from hydra import compose, initialize
import hydra

@dataclass
class LoaderConfig:
    csv_path: str = './'
    spatial_size: list = field(default_factory=lambda: [128,128,128])

class MyDataset(Dataset):
    def __init__(self,csv_path, spatial_size):
        self.csv_path = csv_path
        self.spatial_size = spatial_size

    def __len__(self):
        return 10

    def __getitem__(self, idx):
        return torch.randn(self.spatial_size)


@hydra.main(config_name = 'loader_config')
def main(cfg) -> None:
    print(cfg)
    my_dataset = MyDataset(
        csv_path = cfg.csv_path,
        spatial_size = cfg.spatial_size
    )
    my_dataloader = DataLoader(
        my_dataset,batch_size=1,num_workers=0,shuffle=False
    )

if __name__=='__main__':
    cs = ConfigStore()
    cs.store(name='loader_config',node=LoaderConfig)
    with initialize(version_base=None):
        cfg = compose(config_name='loader_config',overrides=[])
    main(cfg)
  1. run the following in the terminal python monai_hydra.py

Note that changing the above to have the following in the excutable path makes the above run fine.

  if __name__=='__main__':
      # cs = ConfigStore()
      # cs.store(name='loader_config',node=LoaderConfig)
      # with initialize(version_base=None):
      #     cfg = compose(config_name='loader_config',overrides=[])
      cfg = LoaderConfig()
      main(cfg)

Expected behavior
The dataloader should initialise without any issues.

Environment

Ensuring you use the relevant python executable, please paste the output of:

python -c "import monai; monai.config.print_debug_info()"

/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/utils/tf32.py:66: UserWarning: torch.backends.cuda.matmul.allow_tf32 = True by default.
  This value defaults to True when PyTorch version in [1.7, 1.11] and may affect precision.
  See https://docs.monai.io/en/latest/precision_accelerating.html#precision-and-accelerating
  warnings.warn(
================================
Printing MONAI config...
================================
MONAI version: 1.3.1
Numpy version: 1.24.3
Pytorch version: 1.11.0
MONAI flags: HAS_EXT = False, USE_COMPILED = False, USE_META_DICT = False
MONAI rev id: 96bfda00c6bd290297f5e3514ea227c6be4d08b4
MONAI __file__: /home/<username>/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/__init__.py

Optional dependencies:
Pytorch Ignite version: NOT INSTALLED or UNKNOWN VERSION.
ITK version: NOT INSTALLED or UNKNOWN VERSION.
Nibabel version: NOT INSTALLED or UNKNOWN VERSION.
scikit-image version: 0.20.0
scipy version: 1.9.1
Pillow version: 9.4.0
Tensorboard version: 2.13.0
gdown version: NOT INSTALLED or UNKNOWN VERSION.
TorchVision version: 0.12.0
tqdm version: 4.65.0
lmdb version: NOT INSTALLED or UNKNOWN VERSION.
psutil version: 5.9.8
pandas version: 1.5.3
einops version: 0.3.0
transformers version: 4.19.2
mlflow version: NOT INSTALLED or UNKNOWN VERSION.
pynrrd version: NOT INSTALLED or UNKNOWN VERSION.
clearml version: NOT INSTALLED or UNKNOWN VERSION.

For details about installing the optional dependencies, please visit:
    https://docs.monai.io/en/latest/installation.html#installing-the-recommended-dependencies


================================
Printing system config...
================================
System: Linux
Linux version: Ubuntu 22.04.4 LTS
Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.10
Processor: x86_64
Machine: x86_64
Python version: 3.8.5
Process name: python
Command: ['python', '-c', 'import monai; monai.config.print_debug_info()']
Open files: []
Num physical CPUs: 6
Num logical CPUs: 12
Num usable CPUs: 12
CPU usage (%): [8.0, 7.5, 9.3, 6.8, 6.2, 77.2, 8.6, 8.0, 8.1, 9.2, 9.9, 28.8]
CPU freq. (MHz): 3657
Load avg. in last 1, 5, 15 mins (%): [5.4, 5.6, 4.4]
Disk usage (%): 79.3
Avg. sensor temp. (Celsius): UNKNOWN for given OS
Total physical memory (GB): 15.3
Available memory (GB): 6.9
Used memory (GB): 8.0

================================
Printing GPU config...
================================
Num GPUs: 1
Has CUDA: True
CUDA version: 11.3
cuDNN enabled: True
NVIDIA_TF32_OVERRIDE: None
TORCH_ALLOW_TF32_CUBLAS_OVERRIDE: None
cuDNN version: 8200
Current device: 0
Library compiled for CUDA architectures: ['sm_37', 'sm_50', 'sm_60', 'sm_61', 'sm_70', 'sm_75', 'sm_80', 'sm_86', 'compute_37']
GPU 0 Name: NVIDIA GeForce GTX 1650 with Max-Q Design
GPU 0 Is integrated: False
GPU 0 Is multi GPU board: False
GPU 0 Multi processor count: 16
GPU 0 Total memory (GB): 3.8
GPU 0 CUDA capability (maj.min): 7.5

Also using hydra-core 1.3.2

Full trace

  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/hydra/main.py", line 83, in decorated_main
    return task_function(cfg_passthrough)
  File "monai_hydra.py", line 35, in main
    my_dataloader = DataLoader(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/dataloader.py", line 87, in __init__
    set_rnd(dataset, int(_seed))
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 737, in set_rnd
    seed = set_rnd(obj.__dict__[key], seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/monai/data/utils.py", line 727, in set_rnd
    _seed = set_rnd(item, seed=seed)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 359, in __getattr__
    self._format_and_raise(key=key, value=None, cause=e)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 899, in format_and_raise
    _raise(ex, cause)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 797, in _raise
    raise ex.with_traceback(sys.exc_info()[2])  # set env var OC_CAUSE=1 for full trace
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 351, in __getattr__
    return self._get_impl(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 442, in _get_impl
    node = self._get_child(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 73, in _get_child
    child = self._get_node(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 475, in _get_node
    self._validate_get(key)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 164, in _validate_get
    self._format_and_raise(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/base.py", line 231, in _format_and_raise
    format_and_raise(
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 843, in format_and_raise
    object_type = OmegaConf.get_type(node)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 650, in get_type
    return OmegaConf._get_obj_type(c)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 905, in _get_obj_type
    if is_structured_config(c):
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 448, in is_structured_config
    return is_attr_class(obj) or is_dataclass(obj)
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/site-packages/omegaconf/_utils.py", line 442, in is_attr_class
    if attr is None or isinstance(obj, Node):
  File "/home/adwaye/anaconda3/envs/stable_diffusion_2/lib/python3.8/abc.py", line 98, in __instancecheck__
    return _abc_instancecheck(cls, instance)
RecursionError: maximum recursion depth exceeded in comparison
    full_key: set_random_state
    object_type=LoaderConfig

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported Hydra configuration case with the supplied monai_hydra.py script, then inspect monai/data/dataloader.py and monai/data/utils.py, especially set_rnd. Confirm the recursive traversal involving the Hydra configuration and verify that DataLoader initializes successfully while preserving random-seed handling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.