saltstack / saltstack/salt

[BUG] Custom runners loaded from 'runner_dirs:' defined in master config file fail to run from Reactor.

Open
#61,724 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage Reactor
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
A custom Salt Runner is reliably and successfully run via salt-run <file_name>.<function_name> <kwargs>
This same Runner when called via Reactor will initially work, but will then fail with "weakly-referenced object no longer exists".
It can be made to work again by restarting the salt-master service.

salt-run <file_name>.<function_name> <kwargs> continues to work even after reactor causes the "weakly-referenced object no longer exists" error.

Documentation shows that Runners are valid Reactions: Doc link

Setup
My Runner connects to the REST API of NetBox, an IP address management tool.

/etc/salt/master.d/reactor.conf
- netbox/update:
  - salt://reactor/salt-cloud/update_host_info_in_netbox.sls
salt/reactor/salt-cloud/update_host_info_in_netbox.sls
update-netbox:
  runner.netbox.add_record:
    - args:
      - address: {{ data["data"]['ipaddress'] }}
      - dns_name: {{ data["data"]['hostname'] }}
salt/_runners/netbox/init.py

See attached file: runner.txt

The master is a virtual machine running on KVM.

Salt Version:
          Salt: 3004

Dependency Versions:
          cffi: 1.14.5
      cherrypy: unknown
      dateutil: 2.6.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 2.11.3
       libgit2: Not Installed
      M2Crypto: 0.35.2
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: 2.20
      pycrypto: Not Installed
  pycryptodome: Not Installed
        pygit2: Not Installed
        Python: 3.6.8 (default, Mar 19 2021, 05:13:41)
  python-gnupg: Not Installed
        PyYAML: 5.4.1
         PyZMQ: 19.0.0
         smmap: Not Installed
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: centos 8
        locale: UTF-8
       machine: x86_64
       release: 4.18.0-240.15.1.el8_3.x86_64
        system: Linux
       version: CentOS Linux 8

Steps to Reproduce the behavior
The error can be caused by sending an event.
salt-run event.send netbox/update '{"data" : {"hostname": "my-hostname", "ipaddress":"1.1.1.1"}}'

This will work after a salt-master restart, but consecutive tries will eventually fail with weakly-referenced object no longer exists

Expected behavior
The Runner module seems to be good as it can be run reliably with salt-run.
I think the Reactor system is garbage collecting the Runner module and not reloading it when it gets called again. But I am on the fringe of my knowledge here.

Full error

2022-02-25 03:34:44,442 [salt.loader.lazy :791 ][ERROR   ][3848524] Failed to import runners netbox, this is due most likely to a syntax error:
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 752, in _load_module
    self._reload_submodules(mod)
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 619, in _reload_submodules
    for submodule in submodules:
  File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 615, in <genexpr>
    if isinstance(getattr(mod, sname), mod.__class__)
ReferenceError: weakly-referenced object no longer exists

this is due most likely to a syntax error <-This is misleading, if we look at salt/loader/lazy.py we can see that this is a catch all exception.

except Exception as error:  # pylint: disable=broad-except
            log.error(
                "Failed to import %s %s, this is due most likely to a syntax error:\n",
                self.tag,
                name,
                exc_info=True,
            )
            self.missing_modules[name] = error
            return False

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/loader/lazy.py around _load_module and _reload_submodules, then reproduce the failure by sending the netbox/update event repeatedly with salt-run event.send. Done means a custom runner loaded through runner_dirs can be invoked repeatedly by Reactor without the weak-reference error, and the reported import failure is accurate.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.