kohya-ss / kohya-ss/sd-scripts
Runtimeerror when dataset is too large
Open
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
RuntimeError: unable to mmap 4096 bytes from file : Cannot allocate memory (12)
Is there a fix to this? I tried making num_workers =0 the training can go on but it is very slow.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the reported mmap allocation failure during training with a large dataset, comparing num_workers=0 with the failing worker setting. Trace the data-loading path involved and define done as successful training without the RuntimeError while retaining usable worker performance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100