kohya-ss / kohya-ss/sd-scripts

zero3 train flux, accelerate unwrap_model got tensor(size(0))

Open
#1,782 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

Hi, I am using deepspeed zero3 to fine tune flux model using the script - flux_train_network.py.

```
flux = accelerator.unwrap_model(flux)
print(f"flux - {flux.state_dict()['single_blocks.7.linear1.weight'].shape}")
print(f"flux - {flux.state_dict()['single_blocks.7.linear1.weight'].device}")
```
I got the below result:
![image](https://github.com/user-attachments/assets/9592b35b-7ee4-4d75-ab01-3ee786e34514)

and When I save the trained model, I got this:
![image](https://github.com/user-attachments/assets/94a3377a-308f-4de9-9c73-c35241ed6119)

Here is my zero config:
![image](https://github.com/user-attachments/assets/1f9a7ccd-4209-4d08-b4c2-92d3ca4dc9b9)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with flux_train_network.py and the reported accelerator.unwrap_model(flux) call, then compare the zero configuration with the printed tensor shape and device. Reproduce the model save using the provided configuration and verify that the saved weights no longer contain the reported size(0) tensor.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.