saltstack / saltstack/salt

[BUG] scheduler trigger duplicate jobs due multimaster config

Open
#67,983 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
Salt Scheduler on Windows minion, triggers job twice due it's set up in multimaster.
The job is set to run with queue: True

Here the log shows the duplicate execution, there are two Running scheduled job: almost at the same time separated by miliseconds.

It seems to be erratic, some times after removing the job, using schedule.delete, restarting the minion and adding it again, seems to work, but again erratically, could be all good and a sudden change like modifying the job may cause the issue to trigger again.

2025-04-24 05:50:00,067 [salt.utils.schedule:1762][DEBUG   ][4360] schedule: Job schedule_temp_folder_fix was scheduled with jid_include, adding to cache (jid_include defaults to True)
2025-04-24 05:50:00,067 [salt.utils.schedule:1768][DEBUG   ][4360] schedule: Job schedule_temp_folder_fix was scheduled with a max number of 1
2025-04-24 05:50:00,067 [salt.utils.schedule:1792][INFO    ][4360] Running scheduled job: schedule_temp_folder_fix with jid 20250424125000067332
2025-04-24 05:50:00,681 [salt.utils.process:1145][DEBUG   ][4360] Subprocess Schedule(name=schedule_temp_folder_fix, jid=20250424125000067332) added
2025-04-24 05:50:00,734 [salt.utils.schedule:32  ][TRACE   ][4360] ==== evaluating schedule now None =====
2025-04-24 05:50:00,734 [salt.utils.schedule:1762][DEBUG   ][4360] schedule: Job schedule_temp_folder_fix was scheduled with jid_include, adding to cache (jid_include defaults to True)
2025-04-24 05:50:00,734 [salt.utils.schedule:1768][DEBUG   ][4360] schedule: Job schedule_temp_folder_fix was scheduled with a max number of 1
2025-04-24 05:50:00,734 [salt.utils.schedule:1792][INFO    ][4360] Running scheduled job: schedule_temp_folder_fix with jid 20250424125000734042
2025-04-24 05:50:01,226 [salt.utils.process:1145][DEBUG   ][4360] Subprocess Schedule(name=schedule_temp_folder_fix, jid=20250424125000734042) added




2025-04-24 05:50:10,410 [salt.loaded.int.module.cmdmod:922 ][DEBUG   ][3772] stdout: local:

----------

          ID: write_line

    Function: file.append

        Name: C:\\saltLogging.txt

      Result: True

     Comment: Appended 1 lines

     Started: 05:50:08.827882
2025-04-24 05:50:10,410 [salt.loaded.int.module.cmdmod:1352][DEBUG   ][3772] output: local:

----------

          ID: write_line

    Function: file.append

        Name: C:\\saltLogging.txt

      Result: True

     Comment: Appended 1 lines

     Started: 05:50:08.827882



2025-04-24 05:50:12,304 [salt.loaded.int.module.cmdmod:922 ][DEBUG   ][5480] stdout: local:

----------

          ID: write_line

    Function: file.append

        Name: C:\\saltLogging.txt

      Result: True

     Comment: Appended 1 lines

     Started: 05:50:10.410239

2025-04-24 05:50:12,304 [salt.loaded.int.module.cmdmod:1352][DEBUG   ][5480] output: local:

----------

          ID: write_line

    Function: file.append

        Name: C:\\saltLogging.txt

      Result: True

     Comment: Appended 1 lines

     Started: 05:50:10.410239

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)

Please be as specific as possible and give set-up details.

  • on-prem machine
  • VM (Virtualbox, KVM, etc. please specify)
  • VM running on a cloud service, please be explicit and add details
  • container (Kubernetes, Docker, containerd, etc. please specify)
  • or a combination, please be explicit
  • jails if it is FreeBSD
  • classic packaging
  • onedir packaging
  • used bootstrap to install

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
PASTE HERE

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/utils/schedule.py around the scheduling log entries at lines 1762-1792 and trace how a Windows minion handles scheduled jobs with queue: True under multimaster configuration. Reproduce the duplicate execution with the provided log pattern, then add regression coverage showing that one scheduled job is launched once.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.