lightly-ai / lightly-ai/lightly-train
[BUG] Model Training Crashing Windows
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 116
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 6
Description
### 🧠 Describe the Bug
Possible VRAM Memory Leak in detector model training causes Windows to crash. Training of a large model crashed with only 60 total images and a batch size of 2 on a NVIDIA 3080Ti with 12GB of VRAM. This doesn't happen consistently either, as this is the first time this has happened and I have ran these regression tests weekly for the past few weeks without issue on the same dataset, same kwargs.
### 🔁 Steps to Reproduce
lightly_train.train_object_detection(**train_kwargs)
### 🤖 Environment Details
- OS: Windows 11
- Python version: Python3.12
- Frameworks/Libraries (with versions):
torch 2.11.0+cu128
lightly_train 0.14.3
cuda 12.8
### 📌 Additional Context
Output from WinDbg:
VIDEO_TDR_FAILURE (0x116)
nvlddmkm.sys (NVIDIA driver)
0xc000009a (insufficient resources)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start from the lightly_train.train_object_detection(**train_kwargs) entry point and reproduce the run on Windows 11 with the listed Python, PyTorch, CUDA, GPU, dataset, and batch-size details. Monitor GPU memory and system logs while training to narrow down the intermittent resource failure; done means the same training completes without the Windows crash.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- computer-vision, machine-learning, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100