saltstack / saltstack/salt

memoize sdb module's function for config application

Open
#55,740 13 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

pending-discussion Question
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

I've created custom sdb module that reads KDBX
I use this module in Salt Master installations which contain a lot of sdb://kdbx/... schemes.
During:

  File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 3938, in master_config
    apply_sdb(opts)

Every config's sdb scheme causes separate call to heavy method: _load
I thought that if I add @memoize for _load method that would help the KDBX load once

However during evaluating config the apply_sdb doesn't save the memoize closue, thus adding @memoize doesn't help

  1. Could someone explain me why the memoize doesn't work (I'm not a Python dev)? Is it because of https://github.com/saltstack/salt/blob/master/salt/loader.py#L954 sdb loader returning new Loader instance with every sdb entry in config?
  2. How can I achieve caching of loaded KDBX instance for all consecutive entries in config?
Setup
  1. Add any custom module with heavyweight method, like this custom kdbx
  2. Use it in config file:
somekey1: sdb://kdbx/a/b/c
somekey2: sdb://kdbx/a/b/d
somekey3: sdb://kdbx/a/b/e
Steps to Reproduce Issue

With aforementioned setup, run any runner module. It will take ages to run, e.g. salt-run saltutil.sync_all

Versions Report
Salt Version:
           Salt: 2019.2.2
 
Dependency Versions:
           cffi: 1.13.2
       cherrypy: unknown
       dateutil: 2.5.3
      docker-py: 1.10.6
          gitdb: 2.0.0
      gitpython: 2.1.1
          ioflo: Not Installed
         Jinja2: 2.10.3
        libgit2: 0.28.2
        libnacl: Not Installed
       M2Crypto: 0.35.2
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.4.8
   mysql-python: Not Installed
      pycparser: 2.19
       pycrypto: 3.9.4
   pycryptodome: Not Installed
         pygit2: 0.28.2
         Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
   python-gnupg: Not Installed
         PyYAML: 5.2
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.1
        timelib: Not Installed
        Tornado: 4.4.3
            ZMQ: 4.2.1
 
System Versions:
           dist: debian 9.11 
         locale: UTF-8
        machine: x86_64
        release: 4.9.0-8-amd64
         system: Linux
        version: debian 9.11 

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing apply_sdb in salt/config/init.py and the sdb loader around salt/loader.py line 954, then compare how repeated sdb://kdbx entries create or retain loader state. The change is complete when a heavyweight custom _load operation is reused across consecutive configuration entries, with the repeated-entry setup demonstrating that the KDBX data is loaded only once.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.