saltstack / saltstack/salt

[BUG] Windows 3006.10 minions time out with an error in minion.py and no longer communicate with the master 3006.10 (multimaster) after upgrade from 3005-1.2 with multiprocessing: True in minion config

Open
#67,958 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
3006.10 minions time out with an error in minion.py and no longer communicate with the master 3006.10.

Windows Server 2019
Windows Server 2016 Standard
Windows Server 2022

Please be as specific as possible and give set-up details.

  • on-prem machine - vmware VM
  • cloud machine, AWS

Steps to Reproduce the behavior
upgrade to salt from 3005.1-2 to 3006.10

2025-04-13 13:02:06,101 [salt.utils.process:1004][ERROR   ][2880] An un-handled exception from the multiprocessing process 'ProcessPayload(jid=20250413170104743058)' was caught:
Traceback (most recent call last):
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\utils\process.py", line 999, in wrapped_run_func
    return run_func()
  File "C:\Program Files\Salt Project\Salt\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1927, in _target
    run_func(minion_instance, opts, data)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1921, in run_func
    return Minion._thread_return(minion_instance, opts, data)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 2157, in _thread_return
    minion_instance._return_pub(ret)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 2385, in _return_pub
    ret_val = self._send_req_sync(load, timeout=timeout)
  File "C:\Program Files\Salt Project\Salt\Lib\site-packages\salt\minion.py", line 1650, in _send_req_sync
    raise TimeoutError("Request timed out")
TimeoutError: Request timed out

Expected behavior
No timeouts should occur with default minion config of multiprocessing: True. please refer behavior in 3005.1-2, there never used to be this problem.

Screenshots

Image

Minion Versions Report
Windows server 2019

PS C:\Users\vchoudhury_srv> salt-call --versions
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (heads/main:c504d17, Mar  6 2025, 02:25:38) [MSC v.1943 64 bit (AMD64)]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 25.0.2
        relenv: 0.18.1
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist:
        locale: utf-8
       machine: AMD64
       release: 2016Server
        system: Windows
       version: 2016Server 10.0.14393 SP0 Multiprocessor Free

Windows Server 2016

PS C:\Windows\system32> salt-call --versions
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (heads/main:c504d17, Mar  6 2025, 02:25:38) [MSC v.1943 64 bit (AMD64)]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 25.0.2
        relenv: 0.18.1
         smmap: 4.0.0
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist:
        locale: utf-8
       machine: AMD64
       release: 2019Server
        system: Windows
       version: 2019Server 10.0.17763 SP0 Multiprocessor Free

master versions

[root@rlx8gdcpsamp1v ~]# salt --versions
Salt Version:
          Salt: 3006.10

Python Version:
        Python: 3.10.16 (main, Mar  6 2025, 02:23:15) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.6
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.18.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

Salt Extensions:
        SSEAPE: 8.17.0.6

System Versions:
          dist: rhel 8.10 Ootpa
        locale: utf-8
       machine: x86_64
       release: 4.18.0-553.42.1.el8_10.x86_64
        system: Linux
       version: Red Hat Enterprise Linux 8.10 Ootpa

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the 3005.1-2 to 3006.10 upgrade on a Windows minion with multiprocessing enabled, then inspect salt/minion.py and salt/utils/process.py around the traceback. Done means the upgraded minion communicates with the 3006.10 master without request timeouts across the reported Windows versions.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.