kohya-ss / kohya-ss/sd-scripts

Distributed training multi-nodes: port 7860 already in use, by kohya itself...

Open
#1,048 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

What is the exact setup to make distributed training on more than one node?
my secondary node is already set up as "1" to communicate to my main node's IP, I of course set the IP and the port kohya uses (7860)
On the main node I set up as "0" with the Ip being the same of the main computer, and port.

When running a training, I get the following:
```
Traceback (most recent call last):
File "/home/drigo/kohya_ss/venv/bin/accelerate", line 8, in
sys.exit(main())
File "/home/drigo/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/home/drigo/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command
multi_gpu_launcher(args)
File "/home/drigo/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 636, in multi_gpu_launcher
current_env = prepare_multi_gpu_env(args)
File "/home/drigo/kohya_ss/venv/lib/python3.10/site-packages/accelerate/utils/launch.py", line 131, in prepare_multi_gpu_env
raise ConnectionError(
ConnectionError: Tried to launch distributed communication on port `7860`, but another process is utilizing it. Please specify a different port (such as using the `----main_process_port` flag or specifying a different `main_process_port` in your config file) and rerun your script. To automatically use the next open port (on a single node), you can set this to `0`.
```
Well, of course Kohya is the main process using 7860, is the script supposed to open a new one?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the reported accelerate entry point and site-packages/accelerate/commands/launch.py, especially the multi_gpu_launcher path and its port check. Determine whether the project documents a supported multi-node setup and how the main process port should be configured; done means the documented steps explain the port conflict and provide a reproducible configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, machine-learning
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.