RuntimeError: CUDA error: an illegal memory access was encountered
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
When I run my own program without using mixed precision training, it works well. But when I run it with mixed precision training, I get this message.
Traceback (most recent call last):
File "train_Selective_Net_GoPro.py", line 118, in <module>
main(args)
File "train_Selective_Net_GoPro.py", line 77, in main
scale_loss.backward()
File "/home/grd/miniconda3/envs/torch1.1/lib/python3.7/contextlib.py", line 119, in __exit__
next(self.gen)
File "/home/grd/miniconda3/envs/torch1.1/lib/python3.7/site-packages/apex/amp/handle.py", line 127, in scale_loss
should_skip = False if delay_overflow_check else loss_scaler.update_scale()
File "/home/grd/miniconda3/envs/torch1.1/lib/python3.7/site-packages/apex/amp/scaler.py", line 200, in update_scale
self._has_overflow = self._overflow_buf.item()
RuntimeError: CUDA error: an illegal memory access was encountered
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with train_Selective_Net_GoPro.py around lines 77 and 118, then trace the failure through apex/amp/handle.py and apex/amp/scaler.py at the reported overflow update. Reproduce the difference between ordinary and mixed-precision training and establish a minimal case with the stated environment. Done means a confirmed cause and a documented fix or actionable reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100