RuntimeError: CUDA error: an illegal memory access was encountered
Open
Nobody has claimed this yet.
bug
- Dominant language
- Jupyter Notebook
- Stars
- 9.7k
- Forks
- 1.5k
- PR merge metrics
- No merged PRs in 30d
Description
Search before asking
- I have searched the Roboflow Notebooks issues and found no similar bug report.
Notebook name
how-to-auto-train-yolov8-model-with-autodistill.ipynb
Bug
Train target model - YOLOv8:
Default epochs=50, got error and tried again with epochs=2 and got same error.
New https://pypi.org/project/ultralytics/8.0.189 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.0.81 🚀 Python-3.10.12 torch-2.0.1+cu118 CUDA:0 (Tesla V100-SXM2-16GB, 16151MiB)
/content
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
[<ipython-input-22-8c0827b8627e>](https://localhost:8080/#) in <cell line: 6>()
4
5 target_model = YOLOv8("yolov8n.pt")
----> 6 target_model.train(DATA_YAML_PATH, epochs=2)
7 frames
[/usr/local/lib/python3.10/dist-packages/torch/cuda/random.py](https://localhost:8080/#) in cb()
109 for i in range(device_count()):
110 default_generator = torch.cuda.default_generators[i]
--> 111 default_generator.manual_seed(seed)
112
113 _lazy_call(cb, seed_all=True)
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Environment
- google colab V100 51GB RAM
Minimal Reproducible Example
No response
Additional
No response
Are you willing to submit a PR?
- Yes I'd like to help by submitting a PR!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with notebooks/how-to-auto-train-yolov8-model-with-autodistill.ipynb and the cell calling target_model.train(DATA_YAML_PATH, epochs=2). Reproduce the traceback in the reported Google Colab V100 environment, then identify the failing training configuration or dependency interaction. Done means the notebook trains without the illegal memory access or documents a confirmed limitation and workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- jupyter-notebook, python, pytorch
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100