saltstack / saltstack/salt

[BUG] Minion behind NAT incorrectly shows not connected

Open
#66,295 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
I have a minion on my local network connecting to a cloud-based master. That minion never shows as connected, but all my minions with public IP addresses do show as connected. That is, the local minion doesn't show as connected in SaltGUI, nor show up in the results of salt-run manage.present. However, salt functions still work because it is actually connected.

I did some digging around in the Salt source code and discovered that the problem is in the way Salt determines whether a minion is connected.
It works like this:

  1. Get the list of known minions.
  2. Run ss -ant src <master_publish_port> to get a list of active network connections to the master's port.
  3. Compare the list of active connections with each minion's reported IP addresses (like running ip address on the minion).
  4. If one of the minion's reported IP addresses is in the list of connections, it must be connected.
  5. If not, then it is assumed disconnected.

The problem is that when a minion is behind NAT, it doesn't know its public IP address, and therefore is never connected according to this algorithm. A much better way to do it would be for the Salt master to keep a list of the minions that it received a connection from.
However I am not familiar with ZeroMQ so maybe this is impossible?

Setup

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
salt-run manage.present
is showing

- (all minions except fs2)

Expected behavior
salt-run manage.present
should show

- fs2
- (other minions)

Screenshots
image

Versions Report

Salt Version:
          Salt: 3007.0

Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.7
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.5.2
        PyYAML: 6.0.1
         PyZMQ: 25.1.2
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.3.0
       Tornado: 6.3.3
           ZMQ: 4.3.4

Salt Package Information:
  Package Type: onedir

System Versions:
          dist: debian 12.5 bookworm
        locale: utf-8
       machine: x86_64
       release: 6.1.0-18-amd64
        system: Linux
       version: Debian GNU/Linux 12.5 bookworm

Additional context

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the salt-run manage.present entry point and trace the connection-detection flow described in the issue, including the master's handling of minion connections and the ss comparison. Reproduce the behavior with a minion behind NAT; done means manage.present and SaltGUI report that minion as connected while existing Salt functions continue to work.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.