Lightning-AI / Lightning-AI/pytorch-lightning
AdvancedProfiler: ValueError: Attempting to stop recording an action (run_test_evaluation) which was never started.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## π Bug
If you have multiple Trainers and only one runs test, afterwards AdvancedProfiler crashes. This happens even if you free all Trainers except the one you want to run test on.
I have code that has multiple Trainers, doing a grid search. To each Trainer, I just added AdvancedProfiler.
I find the Trainer with the best_model_score and run test only on the best predictor. I then get this crash:
```
DATALOADER:0 TEST RESULTS
{'test_chroma_acc': 0.05000000074505806,
'test_loss': 1763.811279296875,
'test_pitch_acc': 0.0}
--------------------------------------------------------------------------------
Testing: 100%|ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ| 2/2 [00:00<00:00, 22.07it/s]
0%| | 0/3 [00:24
runner()
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1137, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1062, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/site-packages/click/core.py", line 763, in invoke
return __callback(*args, **kwargs)
File "/Users/joseph/dev/neuralaudio/hear-eval-kit/heareval/predictions/runner.py", line 69, in runner
task_predictions(
File "/Users/joseph/dev/neuralaudio/hear-eval-kit/heareval/predictions/task_predictions.py", line 772, in task_predictions
test_scores = best_trainer.test(ckpt_path="best", test_dataloaders=test_dataloader)
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 706, in test
results = self._run(model)
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 918, in _run
self._dispatch()
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 982, in _dispatch
self.accelerator.start_evaluating(self)
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/accelerators/accelerator.py", line 95, in start_evaluating
self.training_type_plugin.start_evaluating(trainer)
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/plugins/training_type/training_type_plugin.py", line 165, in start_evaluating
self._results = trainer.run_stage()
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 993, in run_stage
return self._run_evaluate()
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/trainer/trainer.py", line 1079, in _run_evaluate
eval_loop_results = self._evaluation_loop.run()
File "/usr/local/Cellar/python@3.9/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/contextlib.py", line 124, in __exit__
next(self.gen)
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/profiler/base.py", line 97, in profile
self.stop(action_name)
File "/usr/local/lib/python3.9/site-packages/pytorch_lightning/profiler/advanced.py", line 70, in stop
raise ValueError(f"Attempting to stop recording an action ({action_name}) which was never started.")
ValueError: Attempting to stop recording an action (run_test_evaluation) which was never started.
```
### To Reproduce
https://colab.research.google.com/drive/1rnd1kmwea5BiAYwXq4WaAg5fa8T6GlUI?usp=sharing
I tried to reproduce but BoringModel is currently broken:
```
# some other options for random data
from pl_bolts.datasets import RandomDataset, DummyDataset, RandomDictDataset
/usr/local/lib/python3.7/dist-packages/pl_bolts/utils/warnings.py:32: UserWarning: You want to use `wandb` which is not installed yet, install it with `pip install wandb`.
f' install it with `pip install {pypi_name}`.' + extra_text
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
in ()
1 # some other options for random data
----> 2 from pl_bolts.datasets import RandomDataset, DummyDataset, RandomDictDataset
5 frames
/usr/local/lib/python3.7/dist-packages/pl_bolts/__init__.py in ()
17 _HTTPS_AWS_HUB = "https://pl-bolts-weights.s3.us-east-2.amazonaws.com"
18
---> 19 from pl_bolts import ( # noqa: E402
20 callbacks,
21 datamodules,
/usr/local/lib/python3.7/dist-packages/pl_bolts/datamodules/__init__.py in ()
3 from pl_bolts.datamodules.cifar10_datamodule import CIFAR10DataModule, TinyCIFAR10DataModule
4 from pl_bolts.datamodules.cityscapes_datamodule import CityscapesDataModule
----> 5 from pl_bolts.datamodules.experience_source import DiscountedExperienceSource, ExperienceSource, ExperienceSourceDataset
6 from pl_bolts.datamodules.fashion_mnist_datamodule import FashionMNISTDataModule
7 from pl_bolts.datamodules.imagenet_datamodule import ImagenetDataModule
/usr/local/lib/python3.7/dist-packages/pl_bolts/datamodules/experience_source.py in ()
22
23
---> 24 class ExperienceSourceDataset(IterableDataset):
25 """
26 Basic experience source dataset. Takes a generate_batch function that returns an iterator.
/usr/local/lib/python3.7/dist-packages/torch/utils/data/_typing.py in __new__(cls, name, bases, namespace, **kwargs)
271 for base in bases:
272 if isinstance(base, _DataPipeMeta):
--> 273 return super().__new__(cls, name, bases, namespace, **kwargs) # type: ignore[call-overload]
274
275 namespace.update({'type': _DEFAULT_TYPE,
/usr/lib/python3.7/abc.py in __new__(mcls, name, bases, namespace, **kwargs)
124 """
125 def __new__(mcls, name, bases, namespace, **kwargs):
--> 126 cls = super().__new__(mcls, name, bases, namespace, **kwargs)
127 _abc_init(cls)
128 return cls
/usr/local/lib/python3.7/dist-packages/torch/utils/data/_typing.py in _dp_init_subclass(sub_cls, *args, **kwargs)
369 return_hint.__origin__ == collections.abc.Iterator)):
370 raise TypeError("Expected 'Iterator' as the return annotation for `__iter__` of {}"
--> 371 ", but found {}".format(sub_cls.__name__, _type_repr(hints['return'])))
372 data_type = return_hint.__args__[0]
373 if not issubtype(data_type, sub_cls.type.param):
TypeError: Expected 'Iterator' as the return annotation for `__iter__` of ExperienceSourceDataset, but found typing.Iterable
```
### Expected behavior
Profiling should continue as expected, and output for test only for the trainer which ran test.
### Environment
* CUDA:
- GPU:
- available: False
- version: None
* Packages:
- numpy: 1.19.5
- pyTorch_debug: False
- pyTorch_version: 1.9.0
- pytorch-lightning: 1.4.1
- tqdm: 4.62.0
* System:
- OS: Darwin
- architecture:
- 64bit
-
- processor: i386
- python: 3.9.6
- version: Darwin Kernel Version 19.6.0: Tue Jun 22 19:49:55 PDT 2021; root:xnu-6153.141.35~1/RELEASE_X86_64
Contributor guide
Research direction
Start with pytorch_lightning/profiler/base.py and pytorch_lightning/profiler/advanced.py, then trace the evaluation entry point shown in the traceback. Reproduce the case with multiple Trainers using AdvancedProfiler and only one running test. Done means testing the selected trainer completes without the run_test_evaluation ValueError and produces output only for that trainer.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- performance
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100