saltstack / saltstack/salt

[BUG] job cache is saved locally even if job_cache is set to False in master config

Open
#66,565 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
The salt master config file field job_cache: False does not apply in the work of the system. The jobs are being saved to the job cache at salt master even if job_cache: False is set up in salt master and salt syndic.

Setup
There are 2 salt masters iabc-saltmaster-1 and oabc-saltmaster-1 , salt syndic abc-saltsyndic and minions (for example minion-123).

Salt masters config (iabc-saltmaster-1 and oabc-saltmaster-1 have same configs)
The salt masters /etc/salt/master - part with returner and job cache:


job_cache: False
ext_job_cache: pgjsonb
returner.pgjsonb.host: <postgres db FQDN>
returner.pgjsonb.sslmode: 'require'
returner.pgjsonb.user: <db username>
returner.pgjsonb.pass:  <db password>
returner.pgjsonb.db: <db name>
returner.pgjsonb.port: <db port>
extension_modules: /opt/salt_extension_modules
eauth_tokens: jwt
keep_jobs: 720
keep_jobs_seconds: 2592000
cachedir: /dev/shm/var/cache/salt/master

schedule:
  clean_old_jobs:
    run_on_start: true
    enabled: true
    function: pgjsonb.clean_old_jobs
    splay: 1800
    hours: 6

presence_events: True

The salt masters /etc/salt/minion

master:
 - <oabc-saltmaster-1 FQDN>
 - <iabc-saltmaster-1 FQDN>
metadata_server_grains: True

Salt syndic config (abc-saltsyndic)
The salt syndic /etc/salt/master - part with returner and job cache:

job_cache: False
ext_job_cache: pgjsonb
returner.pgjsonb.host: <postgres db FQDN>
returner.pgjsonb.sslmode: 'require'
returner.pgjsonb.user: <db username>
returner.pgjsonb.pass: <db password>
returner.pgjsonb.db: <db name>
returner.pgjsonb.port: <db port>
extension_modules: /opt/salt_extension_modules
eauth_tokens: jwt
keep_jobs: 720
keep_jobs_seconds: 2592000
cachedir: /dev/shm/var/cache/salt/master

The salt syndic /etc/salt/minion

master:
  - <iabc-saltmaster-1 FQDN>
  - <oabc-saltmaster-1 FQDN>

The machines are VMs running on cloud service with onedir packaging

Steps to Reproduce the behavior

  1. Find any minion assigned to syndic
root@abc-saltsyndic:~# salt-key -L
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] The functions from module 'key' are being loaded by dir() on the loaded module
[DEBUG   ] LazyLoaded key.output
Accepted Keys:
minion-123
Denied Keys:
Rejected Keys:
  1. Run job in the minion
root@abc-saltsyndic:~# salt 'minion-123' test.ping --async
[DEBUG   ] Configuration file path: /etc/salt/master
[DEBUG   ] Reading configuration from /etc/salt/master
[DEBUG   ] Using cached minion ID from /etc/salt/minion_id: abc-saltsyndic
[DEBUG   ] Missing configuration file: /root/.saltrc
[DEBUG   ] MasterEvent PUB socket URI: /var/run/salt/master/master_event_pub.ipc
[DEBUG   ] MasterEvent PULL socket URI: /var/run/salt/master/master_event_pull.ipc
[DEBUG   ] Closing AsyncReqChannel instance
Executed command with job ID: 20240521124445278489
  1. Check if job cache exists
root@abc-saltsyndic:~# find /dev/shm/var/cache/salt/master/jobs -type f -mmin -1
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/.minions.p
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/.load.p
/dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/jid

root@abc-saltsyndic:~# cat /dev/shm/var/cache/salt/master/jobs/37/3237afb5de707c329da87265ad69686867d1476d691c47110cac50c9354209/jid
20240521124445278489

As a resuktl there is jid file with job ID of the job and in the directory are files .minion.p and .load.p.

Expected behavior
According to the docs (Managing the Job Cache) there should be only jid file for checking for and preventing JID collisions. The .minion.p and `.load.p.
should not be created.

Versions Report

on `iabc-saltmaster-1`
Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.16.0
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: 1.10.6
         gitdb: 4.0.11
     gitpython: 3.1.40
        Jinja2: 3.1.2
       libgit2: 1.3.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: 2.6.1
  pycryptodome: 3.9.8
        pygit2: 1.7.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.5 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-169-generic
        system: Linux
       version: Ubuntu 20.04.5 focal
on `oabc-saltmaster-1`
Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: 1.10.6
         gitdb: 4.0.11
     gitpython: 3.1.40
        Jinja2: 3.1.2
       libgit2: 1.3.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: 2.6.1
  pycryptodome: 3.9.8
        pygit2: 1.7.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.6 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-150-generic
        system: Linux
       version: Ubuntu 20.04.6 focal
on `abc-saltsyndic`
Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.11
     gitpython: 3.1.40
        Jinja2: 3.1.2
       libgit2: 1.3.0
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: 1.7.0
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: 5.0.1
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: ubuntu 20.04.5 focal
        locale: utf-8
       machine: x86_64
       release: 5.4.0-169-generic
        system: Linux
       version: Ubuntu 20.04.5 focal

Additional context
Because of need to obfuscate sensitive data all values in triangular brackets <> have been described and the value is not shared.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked Managing the Job Cache documentation and the job_cache settings in /etc/salt/master, then reproduce with the shown salt 'minion-123' test.ping --async command. Done means the cache retains the jid file needed for collision prevention but does not create .minion.p or .load.p under the configured cachedir when job_cache is False.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.