saltstack / saltstack/salt

[BUG] scheduled job not found when first master in multimaster minion is disconnected/not available

Open
#66,722 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
A schedule job has been created in a multiamster (active/active) minion.

The following occurs when the first master is not available/unreachable,
running schedule.run_job schedule_new_task throws Job schedule_new_task does not exist.
But the job is present by schedule.list command
Somehow the job is tied to the first master ?

PS C:\Users\adrian> salt-call schedule.run_job schedule_new_task -l info
[INFO    ] Master saltmaster-pip could not be reached, trying next master (if any)
[WARNING ] Master ip address changed from 172.21.0.10 to 172.21.0.11
local:
    ----------
    comment:
        Job schedule_new_task does not exist.
    result:
        False


PS C:\Users\adrian> salt-call schedule.list
[INFO    ] Master saltmaster-pip could not be reached, trying next master (if any)
[WARNING ] Master ip address changed from 172.21.0.10 to 172.21.0.11
local:
    schedule:
      schedule_new_task:
        args:
        - utils.reboot_system_module
        enabled: true
        function: state.sls
        jid_include: true
        maxrunning: 1
        name: some.example.statefile
        return_job: true
        returner: rawfile_json
        saved: true
        splay: 10
        when:
        - '2024-07-16 19:56:33'


PS C:\Users\adrian> salt-call schedule.run_job schedule_new_task -l info
[INFO    ] Master saltmaster-pip could not be reached, trying next master (if any)
[WARNING ] Master ip address changed from 172.21.0.10 to 172.21.0.11
local:
    ----------
    comment:
        Job schedule_new_task does not exist.
    result:
        False


# using --local triggers the job, but does not complete succesfully given the first master is unreachable
PS C:\Users\adrian> salt-call --local schedule.run_job schedule_new_task -l info
local:
    ----------
    comment:
        Scheduling Job schedule_new_task on minion.
    result:
        True

Setup
3006.x Multimaster minion (active/active) with scheduled job set
The first master is not available/reachable when the commands are executed

Expected behavior
I'd expect the minion to be able to run the scheduled job, despite the first master not being available

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported salt-call schedule.run_job and schedule.list commands with the first master unavailable in a 3006.x multimaster setup. Trace how schedule.run_job locates the existing job when the connection fails over, and verify that the job runs successfully without --local while preserving the expected behavior shown in the report.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.