saltstack / saltstack/salt

reload_modules: True does not reload module dependencies

Open
#24,925 22 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Core Loader severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Hi,

I would like to use the swiftexecution module to install an application.
For the swift execution module to be enabled, one need to install the python-swiftclient library first.

$ salt --versions-report
           Salt: 2015.5.2-120-g1650233
         Python: 2.7.6 (default, Mar 22 2014, 22:59:56)
         Jinja2: 2.7.2
       M2Crypto: 0.21.1
 msgpack-python: 0.4.6
   msgpack-pure: Not Installed
       pycrypto: 2.6.1
        libnacl: Not Installed
         PyYAML: 3.10
          ioflo: Not Installed
          PyZMQ: 14.0.1
           RAET: Not Installed
            ZMQ: 4.0.4
           Mako: Not Installed
        Tornado: Not Installed

Here is a simple SLS file:

python-swiftclient:
  pip.installed:
    - reload_modules: True

python-keystoneclient:
  pip.installed:
    - reload_modules: True

python-dev:
  pkg.installed

download_application:
  module.run:
    - name: swift.get
    - cont: test_container
    - path: test_app
    - local_file: /tmp/test_app
    - require:
      - pip: python-keystoneclient
      - pip: python-swiftclient

The execution of this SLS file fails at the download_application step because it says that the swift module is not present:

     Changes:
----------
          ID: download_application
    Function: module.run
        Name: swift.get
      Result: False
     Comment: Module function swift.get is not available
     Started: 12:42:28.767301
    Duration: 200.984 ms
     Changes:

In order to make it work, I need to restart the minion to be aware of the new python-swiftclient library being installed, and re-run the SLS file.

By looking at the source code, we can see that the python reload() function is called for the modules/swift.py module, so the module is effectively reloaded.

BUT, its dependencies are not (this is not the behaviour of the reload() function)

We can see at https://github.com/saltstack/salt/blob/develop/salt/modules/swift.py#L46 that swift.py imports salt.utils.openstack.swift that is "evaluated" once, on first import (re-importing a library does not mean the library is re-evaluated)

So at https://github.com/saltstack/salt/blob/develop/salt/utils/openstack/swift.py#L22 the following code is only executed once and the variable HAS_SWIFT is still set to False even after the swift.py module is reloaded because reload() does not reload python dependencies.

# Import Swift client libs
HAS_SWIFT = False
try:
    from swiftclient import client

    HAS_SWIFT = True
except ImportError:
    pass

Other modules like docker.io are not affected by this problem because they directly import third-party libraries and upon reloading, they retry to import these libraries. So installing docker-py with reload_modules: True enables the docker.io execution module without restarting the minion.

Adding a reload(suos) call in the virtual() function fixes the problem but this is not a clean solution.

Questions:

  • Do I install the third party library the right way ?
  • Should I expect that the installation of a third party library is taken into account without restarting the minion ?

Cheers.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with salt/modules/swift.py and salt/utils/openstack/swift.py, then reproduce the SLS sequence using pip.installed with reload_modules: True and the swift.get module call. Compare how dependency availability is evaluated after installation and define completion as making the Swift execution module recognize the installed client without restarting the minion.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.