[BUG] job cache is saved locally even if job_cache is set to False in master config
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
The salt master config file field job_cache: False does not apply in the work of the system. The jobs are being saved to the job cache at salt master even if job_cache: False is set up in salt master and salt syndic.
Setup
There are 2 salt masters iabc-saltmaster-1 and oabc-saltmaster-1 , salt syndic abc-saltsyndic and minions (for example minion-123).
Salt masters config (iabc-saltmaster-1 and oabc-saltmaster-1 have same configs)
The salt masters /etc/salt/master - part with returner and job cache:
job_cache: False
ext_job_cache: pgjsonb
returner.pgjsonb.host: <postgres db FQDN>
returner.pgjsonb.sslmode: 'require'
returner.pgjsonb.user: <db username>
returner.pgjsonb.pass: <db password>
returner.pgjsonb.db: <db name>
returner.pgjsonb.port: <db port>
extension_modules: /opt/salt_extension_modules
eauth_tokens: jwt
keep_jobs: 720
keep_jobs_seconds: 2592000
cachedir: /dev/shm/var/cache/salt/master
schedule:
clean_old_jobs:
run_on_start: true
enabled: true
function: pgjsonb.clean_old_jobs
splay: 1800
hours: 6
presence_events: True
The salt masters /etc/salt/minion
master:
- <oabc-saltmaster-1 FQDN>
- <iabc-saltmaster-1 FQDN>
metadata_server_grains: True
Salt syndic config (abc-saltsyndic)
The salt syndic /etc/salt/master - part with returner and job cache:
job_cache: False
ext_job_cache: pgjsonb
returner.pgjsonb.host: <postgres db FQDN>
returner.pgjsonb.sslmode: 'require'
returner.pgjsonb.user: <db username>
returner.pgjsonb.pass: <db password>
returner.pgjsonb.db: <db name>
returner.pgjsonb.port: <db port>
extension_modules: /opt/salt_extension_modules
eauth_tokens: jwt
keep_jobs: 720
keep_jobs_seconds: 2592000
cachedir: /dev/shm/var/cache/salt/master
The salt syndic /etc/salt/minion
master:
- <iabc-saltmaster-1 FQDN>
- <oabc-saltmaster-1 FQDN>
The machines are VMs running on cloud service with onedir packaging
Steps to Reproduce the behavior
- Find any minion assigned to syndic
root@abc-saltsyndic:~# salt-key -L
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] The functions from module 'key' are being loaded by dir() on the loaded module
[DEBUG ] LazyLoaded key.output
Accepted Keys:
minion-123
Denied Keys:
Rejected Keys:
- Run job in the minion
root@abc-saltsyndic:~# salt 'minion-123' test.ping --async
[DEBUG ] Configuration file path: /etc/salt/master
[DEBUG ] Reading configuration from /etc/salt/master
[DEBUG ] Using cached minion ID from /etc/salt/minion_id: abc-saltsyndic
[DEBUG ] Missing configuration file: /root/.saltrc
[DEBUG ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG ] Closing AsyncReqChannel instance
Executed command with job ID: 20240521124445278489
- Check if job cache exists
root@abc-saltsyndic:~# find /dev/shm/var/cache/salt/master/jobs -type f -mmin -1
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/.minions.p
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/.load.p
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/jid
root@abc-saltsyndic:~# cat /dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/jid
20240521124445278489
As a resuktl there is jid file with job ID of the job and in the directory are files .minion.p and .load.p.
Expected behavior
According to the docs (Managing the Job Cache) there should be only jid file for checking for and preventing JID collisions. The .minion.p and `.load.p.
should not be created.
Versions Report
on `iabc-saltmaster-1`
Salt Version:
Salt: 3006.5
Python Version:
Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: unknown
dateutil: 2.8.1
docker-py: 1.10.6
gitdb: 4.0.11
gitpython: 3.1.40
Jinja2: 3.1.2
libgit2: 1.3.0
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: 2.6.1
pycryptodome: 3.9.8
pygit2: 1.7.0
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.14.2
smmap: 5.0.1
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 20.04.5 focal
locale: utf-8
machine: x86_64
release: 5.4.0-169-generic
system: Linux
version: Ubuntu 20.04.5 focal
on `oabc-saltmaster-1`
Salt Version:
Salt: 3006.5
Python Version:
Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: 1.10.6
gitdb: 4.0.11
gitpython: 3.1.40
Jinja2: 3.1.2
libgit2: 1.3.0
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: 2.6.1
pycryptodome: 3.9.8
pygit2: 1.7.0
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.14.2
smmap: 5.0.1
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 20.04.6 focal
locale: utf-8
machine: x86_64
release: 5.4.0-150-generic
system: Linux
version: Ubuntu 20.04.6 focal
on `abc-saltsyndic`
Salt Version:
Salt: 3006.5
Python Version:
Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: unknown
dateutil: 2.8.1
docker-py: Not Installed
gitdb: 4.0.11
gitpython: 3.1.40
Jinja2: 3.1.2
libgit2: 1.3.0
looseversion: 1.0.2
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.9.8
pygit2: 1.7.0
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.14.2
smmap: 5.0.1
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 20.04.5 focal
locale: utf-8
machine: x86_64
release: 5.4.0-169-generic
system: Linux
version: Ubuntu 20.04.5 focal
Additional context
Because of need to obfuscate sensitive data all values in triangular brackets <> have been described and the value is not shared.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Managing the Job Cache documentation and the job_cache settings in /etc/salt/master, then reproduce with the shown salt 'minion-123' test.ping --async command. Done means the cache retains the jid file needed for collision prevention but does not create .minion.p or .load.p under the configured cachedir when job_cache is False.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100