saltstack / saltstack/salt

[BUG] setting source_address doesn't work if default route doesn't function

Open
#58,402 6 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
If you try and work around a non-functional default route by setting the source_address to something that will get matched by policy routing into a functional routing table, the salt-minion will never try to actually connect with the defined source address. In my case I have a local DNS resolver (unbound) that can resolve the DNS name even though the default route is non-functional

Setup

# ip rule
0:	from all lookup local
100:	from 10.87.100.177 lookup 100
32766:	from all lookup main
32767:	from all lookup default
# ip route show table 100
default via 10.87.100.1 dev eth1 proto static
10.87.100.0/24 dev eth1 proto static scope link
# ip route show table main
default via 10.87.100.1 dev eth1
10.87.100.0/24 dev eth1 proto kernel scope link src 10.87.100.249

I have a local unbound DNS resolver that sends DNS requests out specific source addresses so even though the default route doesn't function DNS requests are able to resolve. I imagine the same behaviour could be imitated by setting an entry in /etc/hosts.

Steps to Reproduce the behavior

Finally, set source_address 10.87.100.177 in the salt-minion configuration and restart salt minion, I can see that there is a socket for the minion but it is not for the configured source address

# ss -antp | grep salt-minion
SYN-SENT   0        1                10.87.100.249:38652        10.87.100.5:4506    users:(("salt-minion",pid=707967,fd=15))

The minion never connects.

Expected behavior
With a source_address defined, the minion should try and connect with the source address set in the configuration. Right now it seems in minion.py.resolve_dns it will first try and connect to the master without considering the source_address configuration setting before continuing.

Versions Report

# salt-minion  --versions
Salt Version:
           Salt: 2018.3.4

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: debian 10.4
         locale: utf-8
        machine: x86_64
        release: 5.8.4-200.fc32.x86_64
         system: Linux
        version: debian 10.4

Additional context
If I set my master to an IP address instead of hostname everything functions as expected, so it purely seems to be a problem with how connectivity is checked with the master being a hostname.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in minion.py at resolve_dns and trace how the master hostname is checked before the connection is opened, paying attention to the source_address configuration. Reproduce the setup with a hostname and source_address, then compare it with the working IP-address case. Done means hostname connections use the configured source address and the minion connects through the policy-routed table.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.