Fedora 40 (rawhide) minion cannot maintain connection to master on RHEL 8.

Discussed in https://github.com/saltstack/salt/discussions/65825

^{Originally posted by bchill January 9, 2024}
I have a master running on RHEL 8 and a minion running on Fedora 40.

Both OS are up-to-date and both are running salt rpms:

RHEL 8 master: salt*-3006.5-0.x86_64
Fedora 40 minion: salt*-3006.5-1.fc40.noarch

I have several other minions running on various Ubuntu/RHEL/Rocky/CentOS versions, some ipv4 and some ipv6, some behind home routers and some not - no problems with any of those. This makes me suspect that this is an issue with the 'newness' of fc40 and/or maybe the use of Python 3.12 (used by the rpm - not my doing).

The master accepts the key, but. after that, the connection either cannot be established or keeps dropping as the minion tries repeatedly to connect to the master:

2024-01-09 14:49:52,220 [salt.channel.client:789 ][TRACE   ][12345] Failed to send msg SaltReqTimeoutError('Message timed out')
2024-01-09 14:49:52,220 [salt.channel.client:789 ][TRACE   ][12345] ReqChannel send clear load={'cmd': '_auth', 'id': 'minion.example.com', 'nonce': 'xxxxxx', 'autosign_grains': {'uuid': 'XXXXXXXXXXX'}, 'pub': '-----BEGIN PUBLIC KEY-----\nxxxxxxxx\n-----END PUBLIC KEY-----'}
2024-01-09 14:49:57,226 [salt.channel.client:789 ][TRACE   ][12345] Failed to send msg SaltReqTimeoutError('Message timed out')
2024-01-09 14:49:57,227 [salt.channel.client:789 ][TRACE   ][12345] ReqChannel send clear load={'cmd': '_auth', 'id': 'minion.example.com', 'nonce': 'xxxxxx', 'autosign_grains': {'uuid': 'XXXXXXXXXXX'}, 'pub': '-----BEGIN PUBLIC KEY-----\nxxxxxxxx\n-----END PUBLIC KEY-----'}
2024-01-09 14:50:02,234 [salt.channel.client:789 ][TRACE   ][12345] Failed to send msg SaltReqTimeoutError('Message timed out')
2024-01-09 14:50:02,235 [salt.channel.client:789 ][DEBUG   ][12345] Closing AsyncReqChannel instance
2024-01-09 14:50:02,238 [salt.minion      :789 ][ERROR   ][12345] Error while bringing up minion for multi-master. Is master at master.example.com responding?
2024-01-09 14:50:12,249 [salt.minion      :789 ][DEBUG   ][12345] Connecting to master. Attempt 1 of 1
2024-01-09 14:50:12,251 [salt.utils.network:789 ][DEBUG   ][12345] "master.example.com" Not an IP address? Assuming it is a hostname.
2024-01-09 14:50:12,331 [salt.minion      :789 ][TRACE   ][12345] Custom source interface required: enp2s0f0

nc shows that the master is accessible on both 4505/tcp and 4506/tcp.

ss shows only intermittent connections established to 4506 (on the minion) but never 4505. The 4506 connections last about 30 seconds before disappearing.

The master reports nothing wrong and simply cycles with the minion's connection requests:

2024-01-09 18:39:30,365 [salt.channel.server:289 ][INFO    ][123456] Authentication request from minion.example.com
2024-01-09 18:39:30,366 [salt.utils.user  :23  ][TRACE   ][123456] Trying os.getgrouplist for 'root'
2024-01-09 18:39:30,368 [salt.utils.user  :23  ][TRACE   ][123456] Group list for user 'root': []
2024-01-09 18:39:30,368 [salt.channel.server:547 ][INFO    ][123456] Authentication accepted from minion.example.com
2024-01-09 18:39:30,368 [salt.crypt       :216 ][DEBUG   ][123456] salt.crypt.get_rsa_pub_key: Loading public key
2024-01-09 18:39:30,383 [salt.transport.ipc:23  ][TRACE   ][123456] IPCClient: Connecting to socket: /var/run/salt/master/master_event_pull.ipc
2024-01-09 18:39:30,383 [salt.transport.ipc:23  ][TRACE   ][123453] IPCServer: Handling connection to address: <socket.socket fd=33, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0, laddr=/var/run/salt/master/master_event_pull.ipc>
2024-01-09 18:39:30,384 [salt.utils.event :832 ][DEBUG   ][123456] Sending event: tag = salt/auth; data = {'result': True, 'act': 'accept', 'id': 'minion.example.com', 'pub': '-----BEGIN PUBLIC KEY-----\nxxxxxxxx\n-----END PUBLIC KEY-----', '_stamp': '2024-01-10T00:39:30.384042'}
2024-01-09 18:39:30,385 [salt.crypt       :280 ][DEBUG   ][123456] salt.crypt.get_rsa_key: Loading private key
2024-01-09 18:39:30,385 [salt.crypt       :293 ][DEBUG   ][123456] salt.crypt.sign_message: Signing message.

I really don't know what to look for at this point.

Is this indeed a compatibility bug related to different python versions or some such?

Thanks for any help!

Brian

Start with the reported Salt channel and transport paths: salt.channel.client, salt.channel.server, salt.transport.ipc, and salt.minion, comparing Fedora 40/Python 3.12 with the RHEL 8 master. Reproduce the failed 4505 connection and intermittent 4506 sessions using the logs in the report. Done means the compatibility cause is identified and the minion maintains its connection, with an appropriate regression check if the affected path is found.