saltstack / saltstack/salt

[BUG] Error while bringing up minion for multi-master. Minion unable to successfully connect to a Salt Master.

Open
#66,438 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Minion can't connect to Master, both are 3006.7:

Error while bringing up minion for multi-master. Is master at serverXXX.example.com responding? The error message was Unable to sign_in to master: Attempt to authenticate with the salt master failed with timeout error

Setup
Master is onedir installation (Debian).
Minion is regular installation, based on Fedora CoreOS (FCOS) - not sure if onedir can be used there.

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
Logs:

2024-04-25 15:20:39,673 [salt.cli.daemons :284 ][INFO    ][3591] Starting up the Salt Minion
2024-04-25 15:20:39,674 [salt.utils.event :284 ][INFO    ][3591] Starting pull socket on /var/run/salt/minion/minion_event_b375127e98_pull.ipc
2024-04-25 15:20:39,928 [salt.minion      :284 ][INFO    ][3591] Creating minion process manager
2024-04-25 15:21:15,079 [salt.minion      :284 ][ERROR   ][3591] Error while bringing up minion for multi-master. Is master at serverXXX.example.com responding? The error message was Unable to sign_in to master: Attempt to authenticate with the salt master failed with timeout error
2024-04-25 15:21:39,934 [salt.minion      :284 ][ERROR   ][3591] Minion unable to successfully connect to a Salt Master.

Not a firewall/network issue, salt-master ports are available from minion:

telnet serverXXX.example.com 4505
Trying XX.XX.XX.XX...
Connected to serverXXX.example.com.
Escape character is '^]'.
quit

telnet serverXXX.example.com 4506
Trying XX.XX.XX.XX...
Connected to serverXXX.example.com.
Escape character is '^]'.
quit

Expected behavior
Minion should be able to connect to Master.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
Master:
salt --versions-report
Salt Version:
          Salt: 3006.7

Python Version:
        Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.15.1
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: debian 11 bullseye
        locale: utf-8
       machine: x86_64
       release: 5.10.0-26-amd64
        system: Linux
       version: Debian GNU/Linux 11 bullseye



Minion:
salt-call --versions-report
/usr/lib/python3.12/site-packages/salt/ext/tornado/util.py:246: SyntaxWarning: invalid escape sequence '\d'
  """Unescape a string escaped by `re.escape`.
Salt Version:
          Salt: 3006.7

Python Version:
        Python: 3.12.2 (main, Feb 21 2024, 00:00:00) [GCC 13.2.1 20231205 (Red Hat 13.2.1-6)]

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.2
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.3
       libgit2: Not Installed
  looseversion: 1.3.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.5
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 23.1
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.20.0
        pygit2: Not Installed
  python-gnupg: Not Installed
        PyYAML: 6.0.1
         PyZMQ: 25.1.0
        relenv: Not Installed
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 6.3.3
           ZMQ: 4.3.4

System Versions:
          dist: fedora 39.20240407.3.0
        locale: utf-8
       machine: x86_64
       release: 6.8.4-200.fc39.x86_64
        system: Linux
       version: Fedora Linux 39.20240407.3.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the multi-master connection failure in the provided minion logs, then compare the master and minion versions reports and the successful TCP checks on ports 4505 and 4506. Determine why authentication times out despite reachable Salt Master ports; done means the Fedora CoreOS minion connects successfully to the Debian master.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.