[BUG] Custom runners loaded from 'runner_dirs:' defined in master config file fail to run from Reactor.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
A custom Salt Runner is reliably and successfully run via salt-run <file_name>.<function_name> <kwargs>
This same Runner when called via Reactor will initially work, but will then fail with "weakly-referenced object no longer exists".
It can be made to work again by restarting the salt-master service.
salt-run <file_name>.<function_name> <kwargs> continues to work even after reactor causes the "weakly-referenced object no longer exists" error.
Documentation shows that Runners are valid Reactions: Doc link
Setup
My Runner connects to the REST API of NetBox, an IP address management tool.
/etc/salt/master.d/reactor.conf
- netbox/update:
- salt://reactor/salt-cloud/update_host_info_in_netbox.sls
salt/reactor/salt-cloud/update_host_info_in_netbox.sls
update-netbox:
runner.netbox.add_record:
- args:
- address: {{ data["data"]['ipaddress'] }}
- dns_name: {{ data["data"]['hostname'] }}
salt/_runners/netbox/init.py
See attached file: runner.txt
The master is a virtual machine running on KVM.
Salt Version:
Salt: 3004
Dependency Versions:
cffi: 1.14.5
cherrypy: unknown
dateutil: 2.6.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.3
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: Not Installed
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.6.8 (default, Mar 19 2021, 05:13:41)
python-gnupg: Not Installed
PyYAML: 5.4.1
PyZMQ: 19.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: centos 8
locale: UTF-8
machine: x86_64
release: 4.18.0-240.15.1.el8_3.x86_64
system: Linux
version: CentOS Linux 8
Steps to Reproduce the behavior
The error can be caused by sending an event.
salt-run event.send netbox/update '{"data" : {"hostname": "my-hostname", "ipaddress":"1.1.1.1"}}'
This will work after a salt-master restart, but consecutive tries will eventually fail with weakly-referenced object no longer exists
Expected behavior
The Runner module seems to be good as it can be run reliably with salt-run.
I think the Reactor system is garbage collecting the Runner module and not reloading it when it gets called again. But I am on the fringe of my knowledge here.
Full error
2022-02-25 03:34:44,442 [salt.loader.lazy :791 ][ERROR ][3848524] Failed to import runners netbox, this is due most likely to a syntax error:
Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 752, in _load_module
self._reload_submodules(mod)
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 619, in _reload_submodules
for submodule in submodules:
File "/usr/lib/python3.6/site-packages/salt/loader/lazy.py", line 615, in <genexpr>
if isinstance(getattr(mod, sname), mod.__class__)
ReferenceError: weakly-referenced object no longer exists
this is due most likely to a syntax error <-This is misleading, if we look at salt/loader/lazy.py we can see that this is a catch all exception.
except Exception as error: # pylint: disable=broad-except
log.error(
"Failed to import %s %s, this is due most likely to a syntax error:\n",
self.tag,
name,
exc_info=True,
)
self.missing_modules[name] = error
return False
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt/loader/lazy.py around _load_module and _reload_submodules, then reproduce the failure by sending the netbox/update event repeatedly with salt-run event.send. Done means a custom runner loaded through runner_dirs can be invoked repeatedly by Reactor without the weak-reference error, and the reported import failure is accurate.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100