[BUG] Minion behind NAT incorrectly shows not connected
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
I have a minion on my local network connecting to a cloud-based master. That minion never shows as connected, but all my minions with public IP addresses do show as connected. That is, the local minion doesn't show as connected in SaltGUI, nor show up in the results of salt-run manage.present. However, salt functions still work because it is actually connected.
I did some digging around in the Salt source code and discovered that the problem is in the way Salt determines whether a minion is connected.
It works like this:
- Get the list of known minions.
- Run
ss -ant src <master_publish_port>to get a list of active network connections to the master's port. - Compare the list of active connections with each minion's reported IP addresses (like running
ip addresson the minion). - If one of the minion's reported IP addresses is in the list of connections, it must be connected.
- If not, then it is assumed disconnected.
The problem is that when a minion is behind NAT, it doesn't know its public IP address, and therefore is never connected according to this algorithm. A much better way to do it would be for the Salt master to keep a list of the minions that it received a connection from.
However I am not familiar with ZeroMQ so maybe this is impossible?
Setup
Please be as specific as possible and give set-up details.
- on-prem machine
- VM (Virtualbox, KVM, etc. please specify)
- VM running on a cloud service, please be explicit and add details
- container (Kubernetes, Docker, containerd, etc. please specify)
- or a combination, please be explicit
- jails if it is FreeBSD
- classic packaging
- onedir packaging
- used bootstrap to install
Steps to Reproduce the behavior
salt-run manage.present
is showing
- (all minions except fs2)
Expected behavior
salt-run manage.present
should show
- fs2
- (other minions)
Screenshots
Versions Report
Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: debian 12.5 bookworm
locale: utf-8
machine: x86_64
release: 6.1.0-18-amd64
system: Linux
version: Debian GNU/Linux 12.5 bookworm
Additional context
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the salt-run manage.present entry point and trace the connection-detection flow described in the issue, including the master's handling of minion connections and the ss comparison. Reproduce the behavior with a minion behind NAT; done means manage.present and SaltGUI report that minion as connected while existing Salt functions continue to work.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100