saltstack / saltstack/salt

Proxy Minion does not return schedule result

Open
#62,995 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage Proxy-Minion
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
I believe this is related to issue #39668 which was closed in 2017.
When implementing a schedule with a Salt Proxy Minion, the result of the scheduled task is not returned to the master or to the Event Bus.

Setup
x1 Salt Master running on a Cloud VM with 8GB RAM and 4 vCPU's
x5 Syndic Masters running on a Cloud VM with 256GB RAM and 64 vCPU's serving 1500 proxy minions each.

Steps to Reproduce the behavior
Create a schedule that targets a proxy minion:

c73b4f53-2528-4ff9-af51-82837bfd18e2:
    schedule:
      enabled: true
      example:
        cron: 0 * * * *
        enabled: true
        function: test.ping
        name: example
      switchport_utilization:
        cron: 0 10,14 * * 1-5
        enabled: true
        function: interface.utilization_summary
        name: switchport_utilization
        splay: 1800

Wait for the job to be executed. Below is the output from the proxy debug:

[DEBUG   ] LazyLoaded interface.utilization_summary
[DEBUG   ] LazyLoaded test.ping
[DEBUG   ] LazyLoaded mine.update
[DEBUG   ] Subprocess Thread-263-Schedule-__proxy_keepalive cleaned up
[DEBUG   ] schedule: Job __proxy_keepalive was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG   ] schedule: Job __proxy_keepalive was scheduled with a max number of 1
[INFO    ] Running scheduled job: __proxy_keepalive with jid 20221101172955648942
[DEBUG   ] Subprocess Thread-264-Schedule-__proxy_keepalive added
[DEBUG   ] Using importlib_metadata to load entry points
[DEBUG   ] LazyLoaded status.proxy_reconnect
[DEBUG   ] schedule.handle_func: adding this job to the jobcache with data {'id': 'c73b4f53-2528-4ff9-af51-82837bfd18e2', 'fun': 'status.proxy_reconnect', 'fun_args': [{'proxy_name': 'napalm'}], 'schedule': '__proxy_keepalive', 'jid': '20221101172955648942', 'pid': 11939}
[DEBUG   ] schedule.handle_func: Removing /var/cache/salt/proxy/c73b4f53-2528-4ff9-af51-82837bfd18e2/proc/20221101172955648942
[DEBUG   ] LazyLoaded interface.utilization_summary
[DEBUG   ] LazyLoaded test.ping
[DEBUG   ] LazyLoaded mine.update
[DEBUG   ] Subprocess Thread-264-Schedule-__proxy_keepalive cleaned up

Check the Event Bus.. There is no event for __schedule_return:

20221101173104365994    {
    "_stamp": "2022-11-01T17:31:04.366385",
    "minions": [
        "c73b4f53-2528-4ff9-af51-82837bfd18e2"
    ]
}
salt/job/20221101173104365994/new       {
    "_stamp": "2022-11-01T17:31:04.367102",
    "arg": [],
    "fun": "schedule.list",
    "jid": "20221101173104365994",
    "minions": [
        "c73b4f53-2528-4ff9-af51-82837bfd18e2"
    ],
    "missing": [],
    "tgt": "*",
    "tgt_type": "glob",
    "user": "root"
}
salt/stats/MWorker-3    {
    "_stamp": "2022-11-01T17:31:04.372164",
    "stats": {
        "publish": {
            "mean": 0.006384372711181641,
            "runs": 1
        }
    },
    "time": 1014.6155388355255,
    "worker": "MWorker-3"
}
salt/job/20221101173104365994/ret/c73b4f53-2528-4ff9-af51-82837bfd18e2 {
    "_stamp": "2022-11-01T17:31:04.448417",
    "cmd": "_return",
    "fun": "schedule.list",
    "fun_args": [],
    "id": "c73b4f53-2528-4ff9-af51-82837bfd18e2",
    "jid": "20221101173104365994",
    "retcode": 0,
    "return": "schedule:\n  enabled: true\n  example:\n    cron: 0 * * * *\n    enabled: true\n    function: test.ping\n    name: example\n  switchport_utilization:\n    cron: 0 10,14 * * 1-5\n    enabled: true\n    function: interface.utilization_summary\n    name: switchport_utilization\n    splay: 1800\n",
    "success": true
}
salt/stats/MWorker-0    {
    "_stamp": "2022-11-01T17:31:04.450653",
    "stats": {
        "_return": {
            "mean": 0.002768278121948242,
            "runs": 1
        }
    },
    "time": 1012.1096804141998,
    "worker": "MWorker-0"
}

Check the Job Cache.. There are no jobs for the schedule:

20221101171415775118:
    ----------
    Arguments:
    Function:
        schedule.list
    StartTime:
        2022, Nov 01 17:14:15.775118
    Target:
        *
    Target-type:
        glob
    User:
        root
20221101173104365994:
    ----------
    Arguments:
    Function:
        schedule.list
    StartTime:
        2022, Nov 01 17:31:04.365994
    Target:
        *
    Target-type:
        glob
    User:
        root

Expected behavior
Once a scheduled job is executed, the return is published on the event bus and is exposed to the master.

Versions Report
Salt Version:
Salt: 3005

Dependency Versions:
cffi: 1.14.4
cherrypy: 5.6.0
dateutil: 2.8.1
docker-py: 4.4.4
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.0.2
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 2.6.1
pycryptodome: 3.9.9
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 5.3.1
PyZMQ: 18.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4

System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.76.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No source files or tests are named in the report. Start by tracing proxy minion schedule execution for the shown schedule and the __schedule_return Event Bus path; reproduce the behavior on Salt 3005 and compare it with the expected published return. Done means the scheduled result is published on the Event Bus and exposed to the master.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.