kohya-ss / kohya-ss/sd-scripts

Strange part in calculating the number of steps.

Open
#1,819 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

I found some strange part in flux_train.py to calculate the number of steps.
The number of steps is calculated in this code.
```
args.max_train_steps = args.max_train_epochs * math.ceil(
# len(train_dataloader) / accelerator.num_processes / args.gradient_accumulation_steps
#)

train_dataloader = torch.utils.data.DataLoader(
train_dataset_group,
batch_size=1,
shuffle=True,
collate_fn=collator,
num_workers=n_workers,
persistent_workers=args.persistent_data_loader_workers,
)
```
So, I checked the len(train_dataloader), and I found that batch_size of dataloader is set to 1.
It is strange that no matter what batch size is used during fine-tuning, it is always set to 1
And, if you set this part to the fine-tuning batch size, for example, 4, len(train_dataloader) becomes small and the number of steps calculation becomes strange.

Isn't it correct to change it like the code below?
For example, with the code above, when the number of datasets is 400, the epoch is 100, gradident_accumulation=1, and the batch size is 4, the number of steps is 2500, which is strange, but if you use the modified code below, it becomes 10000 steps.
Isn't it correct that it's 10000 steps?
```
num_examples = train_dataset_group.num_train_images
effective_batch_size = train_batch_size*accelerator.num_processes*args.gradient_accumulation_steps
args.max_train_steps = math.ceil((num_examples*args.max_train_epochs)/effective_batch_size)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Read flux_train.py around the commented max_train_steps calculation and the DataLoader definition, then trace how the configured training batch size, num_train_images, process count, and gradient accumulation are used. Run a fine-tuning configuration matching the issue's 400-image, 100-epoch, batch-size-4 example and verify that the reported step count matches the project's intended training semantics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.