Multi-master config tests all servers even if the first is working
Open
@dwoz is already working on this.
Since Mar 25, 2020.
bug
severity-high
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
I have a multi-master configuration where only my first master can actually be reached (other IPs are only reachable in case the master is failing).
Since I upgraded to 2019.2.2, salt is trying to reach every master even if the first worked well, leading to a VERY long execution (waiting) time.
Setup
/etc/salt/minion.d/minion.conf
master:
- 11.11.11.11 (reachable)
- 22.22.22.22 (unreachable)
- 33.33.33.33 (unreachable)
Adding "master_alive_interval: 10" doesn't seem to reduce the waiting time.
Steps to Reproduce Issue
strace salt-call --log-level=info --output-diff state.apply statetest test=True
[...]
write(2, "[INFO ] Got list of available"..., 106[INFO ] Got list of available master addresses: [u'11.11.11.11', u'22.22.22.22', u'33.33.33.33']
) = 106
stat("/var/log/salt/minion", {st_mode=S_IFREG|0640, st_size=2538887, ...}) = 0
stat("/etc/localtime", {st_mode=S_IFREG|0644, st_size=2971, ...}) = 0
write(3, "2019-10-25 12:38:52,696 [salt.mi"..., 161) = 161
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=100, ...}) = 0
read(7, "## Managed by Salt\n\nsearch xxxxx"..., 512) = 100
read(7, "", 512) = 0
close(7) = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
connect(7, {sa_family=AF_INET, sin_port=htons(4506), sin_addr=inet_addr("11.11.11.11")}, 16) = 0
close(7) = 0
getpid() = 16234
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=100, ...}) = 0
read(7, "## Managed by Salt\n\nsearch xxxxx"..., 512) = 100
read(7, "", 512) = 0
close(7) = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
connect(7, {sa_family=AF_INET, sin_port=htons(4506), sin_addr=inet_addr("22.22.22.22")}, 16
...LONG WAITING TIME... (why is salt trying to reach 22.22.22.22 whereas 11.11.11.11 as reachable?
) = -1 ETIMEDOUT (Connection timed out)
close(7) = 0
getpid() = 16234
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=100, ...}) = 0
read(7, "## Managed by Salt\n\nsearch xxxxx"..., 512) = 100
read(7, "", 512) = 0
close(7) = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
connect(7, {sa_family=AF_INET, sin_port=htons(4506), sin_addr=inet_addr("33.33.33.33")}, 16
...LONG WAITING TIME AGAIN...
) = -1 ETIMEDOUT (Connection timed out)
close(7) = 0
getpid() = 16234
getpid() = 16234
open("/etc/resolv.conf", O_RDONLY|O_CLOEXEC) = 7
fstat(7, {st_mode=S_IFREG|0644, st_size=100, ...}) = 0
read(7, "## Managed by Salt\n\nsearch xxxxx"..., 512) = 100
read(7, "", 512) = 0
close(7) = 0
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 7
connect(7, {sa_family=AF_INET, sin_port=htons(4506), sin_addr=inet_addr("11.11.11.11")}, 16) = 0
close(7) = 0
getpid() = 16234
**HERE BEGINS THE STATE APPLY**
[...]
Versions Report
salt-call --versions-report (minion)
Salt Version:
Salt: 2019.2.2
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.5.3
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.9.4
libgit2: Not Installed
libnacl: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 2.7.13 (default, Sep 26 2018, 18:42:22)
python-gnupg: Not Installed
PyYAML: 3.12
PyZMQ: 16.0.2
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.2.1
System Versions:
dist: debian 9.11
locale: UTF-8
machine: x86_64
release: 4.15.18-19-pve
system: Linux
version: debian 9.11
Master also is running 2019.2.2.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.