saltstack / saltstack/salt

[BUG] schedule splay and maxrunning are ignored when using an inverted range

Open
#65,336 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
The goal is to run a highstate on my minions during night, and I made a schedule for this in my pillar data. However, when too many minions run highstates (in particular apt-get update), my apt-cacher server gets a bad day and hangs the jobs. My workaround solution is to add a splay and maxrunning to the salt schedule, but the next morning all jobs are hung, and they all started at the same second.

Setup

schedule:
  highstate:
    function: state.highstate
    minutes: 60
    splay:
      start: 300
      end: 3600
    maxrunning: 5
    jid_include: True
    run_on_start: False
    range:
      invert: True
      start: '05:00'
      end:   '18:00'

The salt-master is on an LXC VM, the minions are on-prem machines, all running debian 12 using the bullseye onedir.

Expected behavior
Not all jobs firing at the exact same time

Versions Report

(master 3006.3) salt --versions-report 
Salt Version:
          Salt: 3006.3
 
Python Version:
        Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.10
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: debian 12 bookworm
        locale: utf-8
       machine: x86_64
       release: 5.10.0-11-amd64
        system: Linux
       version: Debian GNU/Linux 12 bookworm
(minion 3006.2) salt minion1 test.versions 
Salt Version:
              Salt: 3006.2
     
    Python Version:
            Python: 3.10.12 (main, Aug  3 2023, 21:47:10) [GCC 11.2.0]
     
    Dependency Versions:
              cffi: 1.14.6
          cherrypy: 18.6.1
          dateutil: 2.8.1
         docker-py: Not Installed
             gitdb: Not Installed
         gitpython: Not Installed
            Jinja2: 3.1.2
           libgit2: Not Installed
      looseversion: 1.0.2
          M2Crypto: Not Installed
              Mako: Not Installed
           msgpack: 1.0.2
      msgpack-pure: Not Installed
      mysql-python: Not Installed
         packaging: 22.0
         pycparser: 2.21
          pycrypto: Not Installed
      pycryptodome: 3.9.8
            pygit2: Not Installed
      python-gnupg: 0.4.8
            PyYAML: 6.0.1
             PyZMQ: 23.2.0
            relenv: 0.13.3
             smmap: Not Installed
           timelib: 0.2.4
           Tornado: 4.5.3
               ZMQ: 4.3.4
     
    System Versions:
              dist: debian 12 bookworm
            locale: utf-8
           machine: x86_64
           release: 6.1.0-11-amd64
            system: Linux
           version: Debian GNU/Linux 12 bookworm
(minion 3006.3) salt minion2 test.versions 
    Salt Version:
              Salt: 3006.3
     
    Python Version:
            Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]
     
    Dependency Versions:
              cffi: 1.14.6
          cherrypy: 18.6.1
          dateutil: 2.8.1
         docker-py: Not Installed
             gitdb: Not Installed
         gitpython: Not Installed
            Jinja2: 3.1.2
           libgit2: Not Installed
      looseversion: 1.0.2
          M2Crypto: Not Installed
              Mako: Not Installed
           msgpack: 1.0.2
      msgpack-pure: Not Installed
      mysql-python: Not Installed
         packaging: 22.0
         pycparser: 2.21
          pycrypto: Not Installed
      pycryptodome: 3.9.8
            pygit2: Not Installed
      python-gnupg: 0.4.8
            PyYAML: 6.0.1
             PyZMQ: 23.2.0
            relenv: 0.13.10
             smmap: Not Installed
           timelib: 0.2.4
           Tornado: 4.5.3
               ZMQ: 4.3.4
     
    System Versions:
              dist: debian 12 bookworm
            locale: utf-8
           machine: x86_64
           release: 6.1.0-11-amd64
            system: Linux
           version: Debian GNU/Linux 12 bookworm

Additional context
Yes, I know the real problem is my apt cache server hanging apt update jobs.

EDIT: formatting

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source file or test is named. Start by reproducing the supplied schedule configuration with state.highstate, focusing on the inverted range, splay, and maxrunning options; done means jobs do not all start at the same second and maxrunning continues to limit concurrent jobs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.