[BUG] Windows 3006.10 minions time out with an error in minion.py and no longer communicate with the master 3006.10 (multimaster) after upgrade from 3005-1.2 with multiprocessing: True in minion config
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
3006.10 minions time out with an error in minion.py and no longer communicate with the master 3006.10.
Windows Server 2019
Windows Server 2016 Standard
Windows Server 2022
Please be as specific as possible and give set-up details.
- on-prem machine - vmware VM
- cloud machine, AWS
Steps to Reproduce the behavior
upgrade to salt from 3005.1-2 to 3006.10
2025-04-13 13:02:06,101 [salt.utils.process:1004][ERROR ][2880] An un-handled exception from the multiprocessing process 'ProcessPayload(jid=20250413170104743058)' was caught:
Traceback (most recent call last):
File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\process.py", line 999, in wrapped_run_func
return run_func()
File "C:\Program Files\Salt Project\Salt\Lib\multiprocessing\process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1927, in _target
run_func(minion_instance, opts, data)
File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1921, in run_func
return Minion._thread_return(minion_instance, opts, data)
File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 2157, in _thread_return
minion_instance._return_pub(ret)
File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 2385, in _return_pub
ret_val = self._send_req_sync(load, timeout=timeout)
File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1650, in _send_req_sync
raise TimeoutError("Request timed out")
TimeoutError: Request timed out
Expected behavior
No timeouts should occur with default minion config of multiprocessing: True. please refer behavior in 3005.1-2, there never used to be this problem.
Screenshots
Minion Versions Report
Windows server 2019
PS C:\Users\vchoudhury_srv> salt-call --versions
Salt Version:
Salt: 3006.10
Python Version:
Python: 3.10.16 (heads/main:c504d17, Mar 6 2025, 02:25:38) [MSC v.1943 64 bit (AMD64)]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
cryptography: 42.0.5
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.7
gitpython: Not Installed
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 25.0.2
relenv: 0.18.1
smmap: 4.0.0
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist:
locale: utf-8
machine: AMD64
release: 2016Server
system: Windows
version: 2016Server 10.0.14393 SP0 Multiprocessor Free
Windows Server 2016
PS C:\Windows\system32> salt-call --versions
Salt Version:
Salt: 3006.10
Python Version:
Python: 3.10.16 (heads/main:c504d17, Mar 6 2025, 02:25:38) [MSC v.1943 64 bit (AMD64)]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
cryptography: 42.0.5
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.7
gitpython: Not Installed
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 25.0.2
relenv: 0.18.1
smmap: 4.0.0
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist:
locale: utf-8
machine: AMD64
release: 2019Server
system: Windows
version: 2019Server 10.0.17763 SP0 Multiprocessor Free
master versions
[root@rlx8gdcpsamp1v ~]# salt --versions
Salt Version:
Salt: 3006.10
Python Version:
Python: 3.10.16 (main, Mar 6 2025, 02:23:15) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
cryptography: 42.0.5
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.6
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.18.1
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
Salt Extensions:
SSEAPE: 8.17.0.6
System Versions:
dist: rhel 8.10 Ootpa
locale: utf-8
machine: x86_64
release: 4.18.0-553.42.1.el8_10.x86_64
system: Linux
version: Red Hat Enterprise Linux 8.10 Ootpa
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the 3005.1-2 to 3006.10 upgrade on a Windows minion with multiprocessing enabled, then inspect salt/minion.py and salt/utils/process.py around the traceback. Done means the upgraded minion communicates with the 3006.10 master without request timeouts across the reported Windows versions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100