saltstack / saltstack/salt

[BUG] KeyError: 'master_uri' when calling salt functions from within an engine on a minion

Open
#57,952 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
When attempting to call at least some salt functions from within an engine on a minion (I know pillar.data in particular exhibits this behavior), Salt raises an exception instead of returning a response.

Setup
Minion config at /etc/salt/minion.d/engines.conf:

engines_dirs:
  - /root/salt-distributed-modules/_engines
engines:
  - engine_repro:
      engine_module: "engine_repro"

Engine at /root/salt-distributed-modules/_engines/engine_repro.py:

import time
def start():
    pillar = __salt__['pillar.data']()
    print(pillar)
    time.sleep(10)

Steps to Reproduce the behavior
Create the above files on a minion and restart salt-minion. If you enable debug logging, you'll see:

2020-07-15 18:27:15,667 [salt.minion      :2588][DEBUG   ][20910] Minion of '172.104.19.109' is handling event tag '/salt/minion/minion_schedule_delete_complete'
2020-07-15 18:27:15,667 [salt.minion      :2588][DEBUG   ][20910] Minion of '172.104.19.109' is handling event tag '/salt/minion/minion_schedule_delete_complete'
2020-07-15 18:27:16,667 [salt.utils.schedule:1696][DEBUG   ][20910] schedule: Job __mine_interval was scheduled with jid_include, adding to cache (jid_include defaults to True)
2020-07-15 18:27:16,668 [salt.utils.schedule:1703][DEBUG   ][20910] schedule: Job __mine_interval was scheduled with a max number of 2
2020-07-15 18:27:16,668 [salt.utils.schedule:1721][INFO    ][20910] Running scheduled job: __mine_interval
2020-07-15 18:27:16,672 [salt.utils.process:925 ][DEBUG   ][20910] Subprocess SignalHandlingProcess-1:3-Schedule-__mine_interval added
2020-07-15 18:27:16,693 [salt.utils.schedule:747 ][DEBUG   ][21028] schedule.handle_func: adding this job to the jobcache with data {'id': 'saltminion-dev', 'fun': 'mine.update', 'fun_args': [], 'schedule': '__mine_interval', 'jid': '20200715182716692948', 'pid': 21028}
2020-07-15 18:27:16,696 [salt.utils.lazy  :102 ][DEBUG   ][21028] LazyLoaded mine.update
2020-07-15 18:27:16,701 [salt.utils.lazy  :102 ][DEBUG   ][21028] LazyLoaded config.merge
2020-07-15 18:27:16,702 [salt.utils.schedule:903 ][DEBUG   ][21028] schedule.handle_func: Removing /var/cache/salt/minion/proc/20200715182716692948
2020-07-15 18:27:17,668 [salt.utils.process:934 ][DEBUG   ][20910] Subprocess SignalHandlingProcess-1:3-Schedule-__mine_interval cleaned up
2020-07-15 18:27:25,152 [salt.utils.process:479 ][INFO    ][20910] Process <class 'salt.engines.Engine'> (20976) died with exit status 0, restarting...
2020-07-15 18:27:25,156 [salt.utils.process:460 ][DEBUG   ][20910] Started 'salt.engines.<class 'type'>.Engine' with pid 21036
2020-07-15 18:27:25,166 [salt.utils.lazy  :102 ][DEBUG   ][21036] LazyLoaded engine_repro.start
2020-07-15 18:27:25,169 [salt.utils.lazy  :102 ][DEBUG   ][21036] LazyLoaded pillar.data
2020-07-15 18:27:25,170 [salt.pillar      :79  ][DEBUG   ][21036] Determining pillar cache
2020-07-15 18:27:25,170 [salt.transport.zeromq:173 ][DEBUG   ][21036] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'saltminion-dev', None, 'aes')
2020-07-15 18:27:25,171 [salt.engines     :140 ][CRITICAL][21036] Engine 'engine_repro' could not be started!
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/engines/__init__.py", line 135, in run
    self.engine[self.fun](**kwargs)
  File "/root/salt-distributed-modules/_engines/engine_repro.py", line 3, in start
    pillar = __salt__['pillar.data']()
  File "/usr/lib/python3/dist-packages/salt/modules/pillar.py", line 276, in items
    pillarenv=pillarenv,
  File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 101, in get_pillar
    extra_minion_data=extra_minion_data,
  File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 301, in __init__
    self.channel = salt.transport.client.ReqChannel.factory(opts)
  File "/usr/lib/python3/dist-packages/salt/transport/client.py", line 28, in factory
    AsyncReqChannel.factory, (opts,), kwargs, loop_kwarg="io_loop",
  File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 70, in __init__
    self.obj = cls(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/transport/client.py", line 133, in factory
    return salt.transport.zeromq.AsyncZeroMQReqChannel(opts, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 178, in __new__
    obj.__singleton_init__(opts, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 255, in __singleton_init__
    self.auth = salt.crypt.AsyncAuth(self.opts, io_loop=self._io_loop)
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 491, in __new__
    key = cls.__key(opts)
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 510, in __key
    opts["master_uri"],  # master ID
KeyError: 'master_uri'
2020-07-15 18:27:35,158 [salt.utils.process:479 ][INFO    ][20910] Process <class 'salt.engines.Engine'> (21036) died with exit status 0, restarting...
2020-07-15 18:27:35,165 [salt.utils.process:460 ][DEBUG   ][20910] Started 'salt.engines.<class 'type'>.Engine' with pid 21039
2020-07-15 18:27:35,177 [salt.utils.lazy  :102 ][DEBUG   ][21039] LazyLoaded engine_repro.start
2020-07-15 18:27:35,179 [salt.utils.lazy  :102 ][DEBUG   ][21039] LazyLoaded pillar.data
2020-07-15 18:27:35,179 [salt.pillar      :79  ][DEBUG   ][21039] Determining pillar cache
2020-07-15 18:27:35,179 [salt.transport.zeromq:173 ][DEBUG   ][21039] Initializing new AsyncZeroMQReqChannel for ('/etc/salt/pki/minion', 'saltminion-dev', None, 'aes')
2020-07-15 18:27:35,180 [salt.engines     :140 ][CRITICAL][21039] Engine 'engine_repro' could not be started!
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/salt/engines/__init__.py", line 135, in run
    self.engine[self.fun](**kwargs)
  File "/root/salt-distributed-modules/_engines/engine_repro.py", line 3, in start
    pillar = __salt__['pillar.data']()
  File "/usr/lib/python3/dist-packages/salt/modules/pillar.py", line 276, in items
    pillarenv=pillarenv,
  File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 101, in get_pillar
    extra_minion_data=extra_minion_data,
  File "/usr/lib/python3/dist-packages/salt/pillar/__init__.py", line 301, in __init__
    self.channel = salt.transport.client.ReqChannel.factory(opts)
  File "/usr/lib/python3/dist-packages/salt/transport/client.py", line 28, in factory
    AsyncReqChannel.factory, (opts,), kwargs, loop_kwarg="io_loop",
  File "/usr/lib/python3/dist-packages/salt/utils/asynchronous.py", line 70, in __init__
    self.obj = cls(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/transport/client.py", line 133, in factory
    return salt.transport.zeromq.AsyncZeroMQReqChannel(opts, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 178, in __new__
    obj.__singleton_init__(opts, **kwargs)
  File "/usr/lib/python3/dist-packages/salt/transport/zeromq.py", line 255, in __singleton_init__
    self.auth = salt.crypt.AsyncAuth(self.opts, io_loop=self._io_loop)
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 491, in __new__
    key = cls.__key(opts)
  File "/usr/lib/python3/dist-packages/salt/crypt.py", line 510, in __key
    opts["master_uri"],  # master ID
KeyError: 'master_uri'
2020-07-15 18:27:45,168 [salt.utils.process:479 ][INFO    ][20910] Process <class 'salt.engines.Engine'> (21039) died with exit status 0, restarting...

Expected behavior
Since engines are started by Salt and should have access to e.g. the __salt__ variable, I would have expected __salt__['pillar.data']() in my engine to succeed and return the pillar data for the minion it's running on.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.) 
root@saltminion-dev:/etc/salt/minion.d# salt-minion --versions-report
Salt Version:
           Salt: 3001

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: 3.6.1
         pygit2: Not Installed
         Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: debian 10 buster
         locale: UTF-8
        machine: x86_64
        release: 4.19.0-9-amd64
         system: Linux
        version: Debian GNU/Linux 10 buster

root@saltmaster-dev:~# salt --versions-report
Salt Version:
           Salt: 3001

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: 2.0.5
      gitpython: 2.1.11
         Jinja2: 2.10
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: Not Installed
   pycryptodome: 3.6.1
         pygit2: Not Installed
         Python: 3.7.3 (default, Dec 20 2019, 18:57:59)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 17.1.2
          smmap: 2.0.5
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist: debian 10 buster
         locale: UTF-8
        machine: x86_64
        release: 4.19.0-9-amd64
         system: Linux
        version: Debian GNU/Linux 10 buster

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with salt/engines/init.py and trace how the engine receives options before following the failing call through salt/modules/pillar.py, salt/pillar/init.py, salt/transport/client.py, salt/transport/zeromq.py, and salt/crypt.py. Run the engine_repro.py configuration from the report and verify that calling pillar.data from a minion engine returns pillar data without the KeyError or engine restarts.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.