kohya-ss / kohya-ss/sd-scripts
Distributed training multi-nodes: port 7860 already in use, by kohya itself...
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
What is the exact setup to make distributed training on more than one node?
my secondary node is already set up as "1" to communicate to my main node's IP, I of course set the IP and the port kohya uses (7860)
On the main node I set up as "0" with the Ip being the same of the main computer, and port.
When running a training, I get the following:
```
Traceback (most recent call last):
File "/home/drigo/kohya_ss/venv/bin/accelerate", line 8, in
sys.exit(main())
File "/home/drigo/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/accelerate_cli.py", line 47, in main
args.func(args)
File "/home/drigo/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 977, in launch_command
multi_gpu_launcher(args)
File "/home/drigo/kohya_ss/venv/lib/python3.10/site-packages/accelerate/commands/launch.py", line 636, in multi_gpu_launcher
current_env = prepare_multi_gpu_env(args)
File "/home/drigo/kohya_ss/venv/lib/python3.10/site-packages/accelerate/utils/launch.py", line 131, in prepare_multi_gpu_env
raise ConnectionError(
ConnectionError: Tried to launch distributed communication on port `7860`, but another process is utilizing it. Please specify a different port (such as using the `----main_process_port` flag or specifying a different `main_process_port` in your config file) and rerun your script. To automatically use the next open port (on a single node), you can set this to `0`.
```
Well, of course Kohya is the main process using 7860, is the script supposed to open a new one?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the reported accelerate entry point and site-packages/accelerate/commands/launch.py, especially the multi_gpu_launcher path and its port check. Determine whether the project documents a supported multi-node setup and how the main process port should be configured; done means the documented steps explain the port conflict and provide a reproducible configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, machine-learning
- Issue type
- Documentation
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100