kohya-ss / kohya-ss/sd-scripts

Large datasets produce corrupt LoRAs

Open
#1,380 3 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I am running a 3090Ti on Ubuntu server and I was gearing up to do a large training run on a dataset of half a million images. I began training and it output checkpoints that were corrupt, saying "header is too large' when attempting to load the file. I have found a fix though, adding `--no_metadata` makes it function. this, I believe, confirms my suspicion that the issue is in the reporting of every single tag that the dataset has, and is resulting in too much information being shoved into the metadata. as I don't want to mess with the code myself, I have no way of verifying this suspicion however, I am fairly confident that is what is happening.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file, test, or entry point is identified in the issue. First reproduce a large training run with metadata enabled and compare it with the --no_metadata workaround; done means checkpoints produced from the large dataset load successfully without a "header is too large" error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.