saltstack / saltstack/salt

[BUG] secret not passed to ConnectHandler on status.proxy_reconnect

Open
#62,902 1 comment 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage Proxy-Minion
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
On first run Salt proxy napalm connection is set up successfully. After switch temporarily loses connectivity, is_alive is set to False and status.proxy_reconnect gets called. During reconnect Salt does not pass optional_args: secret to ConnectHandler and reconnect fails.

Setup
Pillar for proxy:

proxy:
  always_alive: true
  driver: brocade_fastiron
  host: **REDACTED**
  username: **REDACTED**
  passwd: **REDACTED**
  multiprocessing: false
  proxytype: napalm
  optional_args:
    secret: **REDACTED**
  • VM (Virtualbox, KVM, etc. please specify)
  • onedir packaging

Steps to Reproduce the behavior

  1. Start proxy minion
  2. Reboot switch
  3. Keepalive sets is_alive to False because it can't reach switch
  4. status.proxy_reconnect gets called but does not pass optional_args: secret to ConnectHandler
  5. Unable to reconnect because optional_args: secret was not passed to ConnectHandler
salt-proxy[3653554]: [ERROR   ] Unhandled exception running status.proxy_reconnect
salt-proxy[3653554]: Traceback (most recent call last):
salt-proxy[3653554]:   File "salt/utils/schedule.py", line 841, in handle_func
salt-proxy[3653554]:     ret["return"] = self.functions[func](*args, **kwargs)
salt-proxy[3653554]:   File "salt/loader/lazy.py", line 149, in __call__
salt-proxy[3653554]:     return self.loader.run(run_func, *args, **kwargs)
salt-proxy[3653554]:   File "salt/loader/lazy.py", line 1228, in run
salt-proxy[3653554]:     return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
salt-proxy[3653554]:   File "salt/loader/lazy.py", line 1243, in _run_as
salt-proxy[3653554]:     return _func_or_method(*args, **kwargs)
salt-proxy[3653554]:   File "/opt/saltstack/salt/run/salt/modules/status.py", line 1814, in proxy_reconnect
salt-proxy[3653554]:     __proxy__[proxy_name + ".init"](opts)  # reopen connection
salt-proxy[3653554]:   File "salt/loader/lazy.py", line 149, in __call__
salt-proxy[3653554]:     return self.loader.run(run_func, *args, **kwargs)
salt-proxy[3653554]:   File "salt/loader/lazy.py", line 1228, in run
salt-proxy[3653554]:     return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
salt-proxy[3653554]:   File "salt/loader/lazy.py", line 1243, in _run_as
salt-proxy[3653554]:     return _func_or_method(*args, **kwargs)
salt-proxy[3653554]:   File "/opt/saltstack/salt/run/salt/proxy/napalm.py", line 200, in init
salt-proxy[3653554]:     "network_device": salt.utils.napalm.get_device(opts),
salt-proxy[3653554]:   File "salt/utils/napalm.py", line 339, in get_device
salt-proxy[3653554]:     network_device["DRIVER"] = _driver_(
salt-proxy[3653554]:   File "/opt/saltstack/salt/pypath/lib/python3.9/site-packages/napalm_brocade_fastiron/brocade_fastiron.py", line 56, in __init__
salt-proxy[3653554]:     'secret': optional_args["secret"],
salt-proxy[3653554]: KeyError: 'secret'

Expected behavior
Reconnect should be successful with switch when status.proxy_reconnect gets called.

Versions Report

Salt Version:
          Salt: 3005.1

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.0
       libgit2: 1.5.0
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.10.1
        Python: 3.9.14 (main, Oct  3 2022, 21:19:16)
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-125-generic
        system: Linux
       version: Ubuntu 20.04 focal

Workaround
We are able to workaround this by adding opts["proxy"]["optional_args"]["secret"] = opts["pillar"]["proxy"]["optional_args"]["secret"] inside if not is_alive statement in /opt/saltstack/salt/run/salt/modules/status.py.

Complete part of this code with workaround:

    if not is_alive:
        opts["proxy"]["optional_args"]["secret"] = opts["pillar"]["proxy"]["optional_args"]["secret"]
        minion_id = opts.get("proxyid", "") or opts.get("id", "")
        log.info("%s (%s proxy) is down. Restarting.", minion_id, proxy_name)
        __proxy__[proxy_name + ".shutdown"](opts)  # safely close connection
        __proxy__[proxy_name + ".init"](opts)  # reopen connection
        log.debug("Restarted %s (%s proxy)!", minion_id, proxy_name)

    return True  # success

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/modules/status.py at proxy_reconnect and trace the opts passed to the proxy init call, then compare with salt/proxy/napalm.py and salt/utils/napalm.py as shown in the traceback. Verify that reconnect receives the configured optional_args secret and that the proxy reconnects successfully after the switch becomes reachable again.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.