saltstack / saltstack/salt

[BUG] Syndics not reading schedule dictionary in minion configuration

Open
#62,454 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage Salt-Syndic
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Salt Syndic minion is not picking up the schedule dictionary in the minion configuration file. This exact dictionary is picked up by every other minion in our environment and results in the state being applied upon minion startup.

Setup

Topology is such that our Salt master manages the syndic , which in turn manages the rest of the minions. (This is a POC at this stage, assessing whether we want to use syndics.)
VMs are running on GCP Compute Engine.

You'll notice that the Salt master and Salt syndic minion configurations are nearly identical - yet, the master runs job1 on startup, the syndic does not.

Salt master (of masters)

/etc/salt/master

"auto_accept": true
"ext_pillar": <REDACTED>
"fileserver_backend":
- "gitfs"
"git_pillar_insecure_auth": true
"git_pillar_password": <REDACTED>
"git_pillar_provider": "pygit2"
"git_pillar_user": <REDACTED>
"gitfs_insecure_auth": true
"gitfs_password": <REDACTED>
"gitfs_provider": "pygit2"
"gitfs_remotes": <REDACTED>
"gitfs_user": <REDACTED>
"order_masters": true
"vault": <REDACTED>

/etc/salt/minion

"acceptance_wait_time": 10
"acceptance_wait_time_max": 80
"auth_safemode": true
"auth_timeout": 60
"grains":
  "env": <REDACTED>
  "region": <REDACTED>
  "role": "saltmaster"
"id": "salt-master"
"key_logfile": "/var/log/salt/key"
"log_file": "/var/log/salt/minion"
"log_level": "info"
"log_level_logfile": "info"
"master":
- "salt-master"
"master_tries": -1
"minion_id_caching": false
"pillarenv": "base"
"ping_interval": 2
"random_reauth_delay": 300
"recon_default": 1000
"recon_max": 60000
"recon_randomize": true
"rejected_retry": true
"retry_dns": 5
"retry_dns_count": 3
"schedule":
  "job1":
    "days": "365"
    "function": "state.apply"
    "maxrunning": 1
    "run_on_start": true
    "splay":
      "end": 600
      "start": 1
  "job2":
    "function": "state.apply"
    "maxrunning": 1
    "run_on_start": false
    "splay":
      "end": 3600
      "start": 1
    "when": "1:00am"

Salt syndic

/etc/salt/master

"auto_accept": true
"ext_pillar": <REDACTED>
"fileserver_backend":
- "gitfs"
"git_pillar_insecure_auth": true
"git_pillar_password": <REDACTED>
"git_pillar_provider": "pygit2"
"git_pillar_user": <REDACTED>
"gitfs_insecure_auth": true
"gitfs_password": <REDACTED>
"gitfs_provider": "pygit2"
"gitfs_remotes": <REDACTED>
"gitfs_user": <REDACTED>
"syndic_master": "salt-master"
"vault": <REDACTED>

/etc/salt/minion

"acceptance_wait_time": 10
"acceptance_wait_time_max": 80
"auth_safemode": true
"auth_timeout": 60
"grains":
  "env": <REDACTED>
  "region": <REDACTED>
  "role": "saltsyndic"
"id": "salt-syndic"
"key_logfile": "/var/log/salt/key"
"log_file": "/var/log/salt/minion"
"log_level": "info"
"log_level_logfile": "info"
"master":
- "salt-master"
"master_tries": -1
"minion_id_caching": false
"pillarenv": "base"
"ping_interval": 2
"random_reauth_delay": 300
"recon_default": 1000
"recon_max": 60000
"recon_randomize": true
"rejected_retry": true
"retry_dns": 5
"retry_dns_count": 3
"schedule":
  "job1":
    "days": "365"
    "function": "state.apply"
    "maxrunning": 1
    "run_on_start": true
    "splay":
      "end": 600
      "start": 1
  "job2":
    "function": "state.apply"
    "maxrunning": 1
    "run_on_start": false
    "splay":
      "end": 3600
      "start": 1
    "when": "1:00am"

Steps to Reproduce the behavior
Reproduction steps are simply to start up the syndic minion and observe that job1 is not triggered on startup.

Expected behavior
The syndic was expected to run job1 on startup.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
Salt Version:
          Salt: 3004.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: unknown
      dateutil: 2.7.3
     docker-py: Not Installed
         gitdb: 2.0.6
     gitpython: 3.0.7
        Jinja2: 2.10.1
       libgit2: 0.28.3
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 0.6.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: 2.6.1
  pycryptodome: 3.6.1
        pygit2: 1.0.3
        Python: 3.8.10 (default, Jun 22 2022, 20:18:18)
  python-gnupg: 0.4.5
        PyYAML: 5.3.1
         PyZMQ: 18.1.1
         smmap: 2.0.5
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.3.2

System Versions:
          dist: ubuntu 20.04 focal
        locale: utf-8
       machine: x86_64
       release: 5.15.0-1013-gcp
        system: Linux
       version: Ubuntu 20.04 focal

Additional Details

Salt master lists expected schedules

sentientglare@salt-master:~$ sudo salt-call schedule.list show_all=True show_disabled=True
local:
    schedule:
      __mine_interval:
        enabled: true
        function: mine.update
        jid_include: true
        maxrunning: 2
        minutes: 60
        name: __mine_interval
        return_job: false
        run_on_start: true
      job1:
        days: '365'
        enabled: true
        function: state.apply
        jid_include: true
        maxrunning: 1
        name: job1
        run_on_start: true
        splay:
          end: 600
          start: 1
      job2:
        enabled: true
        function: state.apply
        jid_include: true
        maxrunning: 1
        name: job2
        run_on_start: false
        splay:
          end: 3600
          start: 1
        when: 1:00am

Salt syndic only lists a default schedule

sentientglare@salt-syndic:~$ sudo salt-call schedule.list show_all=True show_disabled=True
local:
    schedule:
      __mine_interval:
        enabled: true
        function: mine.update
        jid_include: true
        maxrunning: 2
        minutes: 60
        name: __mine_interval
        return_job: false
        run_on_start: true

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing schedule handling for the syndic minion with the regular minion, using the /etc/salt/minion configuration and the salt-call schedule.list show_all=True show_disabled=True output described here. Reproduce the startup behavior with the provided Salt 3004.2 setup; done means the syndic loads job1 and triggers it on startup, while retaining the default schedule entries.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.