kohya-ss / kohya-ss/sd-scripts
Large datasets produce corrupt LoRAs
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I am running a 3090Ti on Ubuntu server and I was gearing up to do a large training run on a dataset of half a million images. I began training and it output checkpoints that were corrupt, saying "header is too large' when attempting to load the file. I have found a fix though, adding `--no_metadata` makes it function. this, I believe, confirms my suspicion that the issue is in the reporting of every single tag that the dataset has, and is resulting in too much information being shoved into the metadata. as I don't want to mess with the code myself, I have no way of verifying this suspicion however, I am fairly confident that is what is happening.
Contributor guide
No contributing guide indexed for this repository
Research direction
No source file, test, or entry point is identified in the issue. First reproduce a large training run with metadata enabled and compare it with the --no_metadata workaround; done means checkpoints produced from the large dataset load successfully without a "header is too large" error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100