saltstack / saltstack/salt

Multi-master config tests all servers even if the first is working

Open
#55,123 7 comments 0 reactions 1 assignee View on GitHub

@dwoz is already working on this.

Since Mar 25, 2020.

bug severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

I have a multi-master configuration where only my first master can actually be reached (other IPs are only reachable in case the master is failing).

Since I upgraded to 2019.2.2, salt is trying to reach every master even if the first worked well, leading to a VERY long execution (waiting) time.

Setup

/etc/salt/minion.d/minion.conf

master:
    - 11.11.11.11 (reachable)
    - 22.22.22.22 (unreachable)
    - 33.33.33.33 (unreachable)

Adding "master_alive_interval: 10" doesn't seem to reduce the waiting time.

Steps to Reproduce Issue

strace salt-call --log-level=info --output-diff state.apply statetest test=True

[...]
write(2, "[INFO    ] Got list of available"..., 106[INFO    ] Got list of available master addresses: [u'11.11.11.11', u'22.22.22.22', u'33.33.33.33']
) = 106
stat("/var/log/salt/minion", {st_mode=S_IFREG|0640, st_size=2538887, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2971, ...}) = 0
write(3, "2019-10-25 12:38:52,696 [salt.mi"..., 161) = 161
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=100, ...}) = 0
read(7, "## Managed by Salt\n\nsearch xxxxx"..., 512) = 100
read(7, "", 512)                        = 0
close(7)                                = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
connect(7, {sa_family=AF_INET, sin_port=htons(4506), sin_addr=inet_addr("11.11.11.11")}, 16) = 0
close(7)                                = 0
getpid()                                = 16234
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=100, ...}) = 0
read(7, "## Managed by Salt\n\nsearch xxxxx"..., 512) = 100
read(7, "", 512)                        = 0
close(7)                                = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
connect(7, {sa_family=AF_INET, sin_port=htons(4506), sin_addr=inet_addr("22.22.22.22")}, 16

...LONG WAITING TIME... (why is salt trying to reach 22.22.22.22 whereas 11.11.11.11 as reachable?

) = -1 ETIMEDOUT (Connection timed out)
close(7)                                = 0
getpid()                                = 16234
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=100, ...}) = 0
read(7, "## Managed by Salt\n\nsearch xxxxx"..., 512) = 100
read(7, "", 512)                        = 0
close(7)                                = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
connect(7, {sa_family=AF_INET, sin_port=htons(4506), sin_addr=inet_addr("33.33.33.33")}, 16

...LONG WAITING TIME AGAIN...

) = -1 ETIMEDOUT (Connection timed out)
close(7)                                = 0
getpid()                                = 16234
getpid()                                = 16234
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=100, ...}) = 0
read(7, "## Managed by Salt\n\nsearch xxxxx"..., 512) = 100
read(7, "", 512)                        = 0
close(7)                                = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
connect(7, {sa_family=AF_INET, sin_port=htons(4506), sin_addr=inet_addr("11.11.11.11")}, 16) = 0
close(7)                                = 0
getpid()                                = 16234

**HERE BEGINS THE STATE APPLY**

[...]
Versions Report

salt-call --versions-report (minion)

Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.5.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.9.4
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
   python-gnupg: Not Installed
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.11 
         locale: UTF-8
        machine: x86_64
        release: 4.15.18-19-pve
         system: Linux
        version: debian 9.11 

Master also is running 2019.2.2.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.