memoize sdb module's function for config application
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description of Issue
I've created custom sdb module that reads KDBX
I use this module in Salt Master installations which contain a lot of sdb://kdbx/... schemes.
During:
File "/usr/lib/python3/dist-packages/salt/config/__init__.py", line 3938, in master_config
apply_sdb(opts)
Every config's sdb scheme causes separate call to heavy method: _load
I thought that if I add @memoize for _load method that would help the KDBX load once
However during evaluating config the apply_sdb doesn't save the memoize closue, thus adding @memoize doesn't help
- Could someone explain me why the memoize doesn't work (I'm not a Python dev)? Is it because of https://github.com/saltstack/salt/blob/master/salt/loader.py#L954 sdb loader returning new Loader instance with every sdb entry in config?
- How can I achieve caching of loaded KDBX instance for all consecutive entries in config?
Setup
- Add any custom module with heavyweight method, like this custom kdbx
- Use it in config file:
somekey1: sdb://kdbx/a/b/c
somekey2: sdb://kdbx/a/b/d
somekey3: sdb://kdbx/a/b/e
Steps to Reproduce Issue
With aforementioned setup, run any runner module. It will take ages to run, e.g. salt-run saltutil.sync_all
Versions Report
Salt Version:
Salt: 2019.2.2
Dependency Versions:
cffi: 1.13.2
cherrypy: unknown
dateutil: 2.5.3
docker-py: 1.10.6
gitdb: 2.0.0
gitpython: 2.1.1
ioflo: Not Installed
Jinja2: 2.10.3
libgit2: 0.28.2
libnacl: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.4.8
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 3.9.4
pycryptodome: Not Installed
pygit2: 0.28.2
Python: 3.5.3 (default, Sep 27 2018, 17:25:39)
python-gnupg: Not Installed
PyYAML: 5.2
PyZMQ: 16.0.2
RAET: Not Installed
smmap: 2.0.1
timelib: Not Installed
Tornado: 4.4.3
ZMQ: 4.2.1
System Versions:
dist: debian 9.11
locale: UTF-8
machine: x86_64
release: 4.9.0-8-amd64
system: Linux
version: debian 9.11
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing apply_sdb in salt/config/init.py and the sdb loader around salt/loader.py line 954, then compare how repeated sdb://kdbx entries create or retain loader state. The change is complete when a heavyweight custom _load operation is reused across consecutive configuration entries, with the repeated-entry setup demonstrating that the KDBX data is loaded only once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100