[BUG] Minion loses the connection after updating from 3004 to 3005 onedir
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
I want to update my environment to 3005, for now I have updated my test master to 3005, all new minions with fresh 3005 installation seem to work correctly, but when I update minions from 3004 to 3005 then they lose the connection
Setup
Minion with 3004 classic version, already in contact with master (master running 3005 onedir), then I perform the update to 3005 onedir, everything seems normal on minion side, no errors, service runs normally, but I'm not able to contact the minion from the master having the error:
Minion did not return. [No response]
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
- upgrade minion from 3004.2 classic to 3005.1 onedir
- run a test.ping
Restarting the minion throws the following errors:
Dec 15 10:51:36 minion01 systemd[1]: Started The Salt Minion.
Dec 15 10:51:40 minion01 salt-minion[292047]: [WARNING ] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x7f28c5a72d90>
Dec 15 10:52:54 minion01 systemd[1]: Stopping The Salt Minion...
Dec 15 10:52:54 minion01 salt-minion[292047]: [WARNING ] Minion received a SIGTERM. Exiting.
Dec 15 10:52:54 minion01 salt-minion[292047]: The Salt Minion is shutdown. Minion received a SIGTERM. Exited.
Dec 15 10:52:54 minion01 salt-minion[292047]: TRACE:salt.transport.ipc:IPCClient: Connecting to socket: /var/run/salt/minion/minion_event_ce6006f7de_pull.ipc
Dec 15 10:52:55 minion01 salt-minion[292047]: TRACE:salt.transport.ipc:IPCClient: Connecting to socket: /var/run/salt/minion/minion_event_ce6006f7de_pull.ipc
Dec 15 10:52:55 minion01 salt-minion[292047]: ERROR:salt.utils.event:Unable to connect pusher: Stream is closed
Dec 15 10:52:55 minion01 salt-minion[292047]: TRACE:salt.minion:ret_val = <salt.ext.tornado.concurrent.Future object at 0x7f28c47bd4c0>
Dec 15 10:52:55 minion01 salt-minion[292047]: TRACE:salt.transport.ipc:Subscriber disconnected from IPC /var/run/salt/minion/minion_event_ce6006f7de_pub.ipc
Dec 15 10:52:55 minion01 salt-minion[292047]: TRACE:salt.transport.ipc:Client disconnected from IPC /var/run/salt/minion/minion_event_ce6006f7de_pull.ipc
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.crypt:salt.crypt.get_rsa_key: Loading private key
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.crypt:Loaded minion key: /etc/salt/pki/minion/minion.pem
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.crypt:salt.crypt.verify_signature: Loading public key
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.crypt:salt.crypt.get_rsa_pub_key: Loading public key
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.crypt:salt.crypt.verify_signature: Verifying signature
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.channel.client:Closing AsyncReqChannel instance
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.minion:Refreshing matchers.
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.transport.ipc:Closing IPCMessageClient instance
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.minion:Refreshing beacons.
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.utils.event:SaltEvent PUB socket URI: /var/run/salt/minion/minion_event_ce6006f7de_pub.ipc
Dec 15 10:52:55 minion01 salt-minion[292047]: DEBUG:salt.utils.event:SaltEvent PULL socket URI: /var/run/salt/minion/minion_event_ce6006f7de_pull.ipc
Dec 15 10:52:55 minion01 salt-minion[292047]: TRACE:salt.transport.ipc:IPCClient: Connecting to socket: /var/run/salt/minion/minion_event_ce6006f7de_pull.ipc
Dec 15 10:52:56 minion01 salt-minion[292047]: TRACE:salt.transport.ipc:IPCClient: Connecting to socket: /var/run/salt/minion/minion_event_ce6006f7de_pull.ipc
Dec 15 10:52:56 minion01 salt-minion[292047]: ERROR:salt.utils.event:Unable to connect pusher: Stream is closed
Dec 15 10:52:56 minion01 salt-minion[292047]: TRACE:salt.channel.client:Failed to send msg SaltReqTimeoutError('Message timed out')
Dec 15 10:52:56 minion01 salt-minion[292047]: TRACE:salt.channel.client:ReqChannel send crypt load={'id': 'minion01.corp.ad.net', 'cmd': '_minion_event', 'pretag': None, 'tok': b'3\x1f1\xd3\x18\xc7Hg\xabS<\x832\x1ct\xdf\xebD\xae:%4\xaf\xbeb\xadcm\xf3\x11\xad\xc2\xe3T\xd2B\xa6\';\xf2\xc3\xa2\x8f\xe5z\x00\xb9\xcd\xfcW>\x0f\xf7U\xf5<e\x97C]\x0f\xb1~\x19x\xd7\xc2\x8e5D\x1e\xc3\xae\xadp\x05`\xf4\xe6\x7f8\xda\xae*b\x88\xd2y\x8f\x0fG!\xd0\xc9:\xba2j)\x1c\xd1\xe47\xfbeo\x96\xffI\xf9\x89<\xc5V\x84J?\x8e\xd0\xf7E\xe4\\\xe3\xa9F\xd3-\xff%\xe4M\x1aET\xbd}\xa60\x88\xb9\x06\x12\x04ul\xfe\xa7\n\xe5\xea\xa0\xcf\x87?\xa48\xday/\x1d\xa7\x8b9\xcab\xdf\xc6N\xf6\xf1\xc8F\xec\xe4\xf9J\xd6\xb8\xa1\x9d "f\x8a0q\xfd\x1b\xc63\x1e\xbe\x87>\x15\x98\x85J\xb0\xb9\xf7\xb1/\x8c|\xaeY\xeb\x81\x91\xd8\x1d\xad\xe0\xc8~\x95\xafB6\x0f\x8b\x97\x86<Y!\x04a\x8d` \xc9\x82~\xc8\xd3\x04\xc7\xee[\xb0H{\xf6\xcc^^2\xcbRc\xb7\xfb\x86', 'data': 'Minion minion01.corp.ad.net started at Thu Dec 15 10:51:40 2022', 'tag': 'minion_start', '_stamp': '2022-12-15T09:51:40.733525', 'nonce': '2edea3c4551a4f889cdd815a3d3a34be'}
Dec 15 10:54:24 minion01 systemd[1]: salt-minion.service: State 'stop-sigterm' timed out. Killing.
Dec 15 10:54:24 minion01 systemd[1]: salt-minion.service: Killing process 292022 (/opt/saltstack/) with signal SIGKILL.
Dec 15 10:54:24 minion01 systemd[1]: salt-minion.service: Main process exited, code=killed, status=9/KILL
Dec 15 10:54:24 minion01 systemd[1]: salt-minion.service: Failed with result 'timeout'.
Dec 15 10:54:24 minion01 systemd[1]: salt-minion.service: Unit process 292047 (/opt/saltstack/) remains running after unit stopped.
Dec 15 10:54:24 minion01 systemd[1]: Stopped The Salt Minion.
Dec 15 10:54:24 minion01 systemd[1]: salt-minion.service: Consumed 12.902s CPU time.
Dec 15 10:54:26 minion01 salt-minion[292047]: ERROR:salt.scripts:Minion process encountered exception: [Errno 3] No such process
Expected behavior
Communication from master to minion remains
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)Salt Version:
Salt: 3005.1
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.0
libgit2: Not Installed
M2Crypto: 0.38.0
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: Not Installed
Python: 3.9.15 (main, Nov 8 2022, 03:42:58)
python-gnupg: 0.4.8
PyYAML: 5.4.1
PyZMQ: 23.2.0
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.04 jammy
locale: utf-8
machine: x86_64
release: 5.15.0-56-generic
system: Linux
version: Ubuntu 22.04 jammy
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the 3004 classic to 3005.1 onedir upgrade and run test.ping against the salt-minion systemd service. Inspect the reported IPC transport and shutdown errors in the service logs, then verify that master-to-minion communication remains available after the upgrade and restart.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, python, ubuntu
- Domain
- devops, infrastructure, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100