roboflow / roboflow/notebooks

RuntimeError: CUDA error: an illegal memory access was encountered

Open
#208 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Jupyter Notebook
Stars
9.7k
Forks
1.5k
PR merge metrics
No merged PRs in 30d

Description

Search before asking
  • I have searched the Roboflow Notebooks issues and found no similar bug report.
Notebook name

how-to-auto-train-yolov8-model-with-autodistill.ipynb

Bug

Train target model - YOLOv8:
Default epochs=50, got error and tried again with epochs=2 and got same error.

New https://pypi.org/project/ultralytics/8.0.189 available 😃 Update with 'pip install -U ultralytics'
Ultralytics YOLOv8.0.81 🚀 Python-3.10.12 torch-2.0.1+cu118 CUDA:0 (Tesla V100-SXM2-16GB, 16151MiB)
/content
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
[<ipython-input-22-8c0827b8627e>](https://localhost:8080/#) in <cell line: 6>()
      4 
      5 target_model = YOLOv8("yolov8n.pt")
----> 6 target_model.train(DATA_YAML_PATH, epochs=2)

7 frames
[/usr/local/lib/python3.10/dist-packages/torch/cuda/random.py](https://localhost:8080/#) in cb()
    109         for i in range(device_count()):
    110             default_generator = torch.cuda.default_generators[i]
--> 111             default_generator.manual_seed(seed)
    112 
    113     _lazy_call(cb, seed_all=True)

RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Environment
  • google colab V100 51GB RAM
Minimal Reproducible Example

No response

Additional

No response

Are you willing to submit a PR?
  • Yes I'd like to help by submitting a PR!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with notebooks/how-to-auto-train-yolov8-model-with-autodistill.ipynb and the cell calling target_model.train(DATA_YAML_PATH, epochs=2). Reproduce the traceback in the reported Google Colab V100 environment, then identify the failing training configuration or dependency interaction. Done means the notebook trains without the illegal memory access or documents a confirmed limitation and workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
jupyter-notebook, python, pytorch
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.