[BUG] LazyLoader._refresh_file_mapping is called for every load miss
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
I'm working on tracing for Salt and I noticed a peculiar thing:
Here's the part of highstate that produced the trace above:
less:
pkg.installed:
- version: latest
- require:
- sls: apt
htop:
pkg.installed:
- version: latest
- require:
- sls: apt
You may notice that pkg.check_db is loaded for both states and it takes 25ms. The same applies to other pkg related functions. The thing is: pkg.check_db is only implemented by salt.modules.ebuildpkg when running on Gentoo, but I'm using salt.modules.aptpkg, since I'm running Debian.
Let's expand the trace for less:
Now it turns out that most of the time is spent in salt.loader.LazyLoader._refresh_file_mapping, which is getting called every time a non-implemented function is requested.
This time really adds up:
- 190ms total to enforce
pkg.installedforless - 65ms to run
apt-cacheand make sure it's already on thelatest(actual work) - 112ms to try load unloadable functions and refresh file mappings 4 times
That's 60% of time spent on fruitless activities.
Expected behavior
Salt doesn't call salt.loader.LazyLoader._refresh_file_mapping repeatedly when there is no reason to do so.
The simplest way for pkg module is not to call Gentoo specific functions and put these calls into salt.modules.ebuildpkg directly. The proper way is to implement negative caching of some sort.
Versions Report
salt --versions-report
Salt Version:
Salt: 3002.2
Dependency Versions:
cffi: Not Installed
cherrypy: Not Installed
dateutil: 2.8.1
docker-py: 4.1.0
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 2.11.2
libgit2: Not Installed
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.0
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: Not Installed
pycrypto: Not Installed
pycryptodome: 3.9.7
pygit2: Not Installed
Python: 3.9.1 (default, Dec 8 2020, 07:51:42)
python-gnupg: Not Installed
PyYAML: 5.3.1
PyZMQ: 20.0.0
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.3.3
System Versions:
dist: debian testing bullseye
locale: utf-8
machine: aarch64
release: 5.11.0-rc2-ivan
system: Linux
version: Debian GNU/Linux testing bullseye
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in salt.loader.LazyLoader._refresh_file_mapping and trace the load-miss path described in the issue, then compare the pkg calls made through salt.modules.aptpkg and salt.modules.ebuildpkg. Reproduce the repeated refreshes with the provided pkg.installed highstate example and verify that missing functions no longer trigger unnecessary file-mapping refreshes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100