Proxy Minion does not return schedule result
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
I believe this is related to issue #39668 which was closed in 2017.
When implementing a schedule with a Salt Proxy Minion, the result of the scheduled task is not returned to the master or to the Event Bus.
Setup
x1 Salt Master running on a Cloud VM with 8GB RAM and 4 vCPU's
x5 Syndic Masters running on a Cloud VM with 256GB RAM and 64 vCPU's serving 1500 proxy minions each.
Steps to Reproduce the behavior
Create a schedule that targets a proxy minion:
c73b4f53-2528-4ff9-af51-82837bfd18e2:
schedule:
enabled: true
example:
cron: 0 * * * *
enabled: true
function: test.ping
name: example
switchport_utilization:
cron: 0 10,14 * * 1-5
enabled: true
function: interface.utilization_summary
name: switchport_utilization
splay: 1800
Wait for the job to be executed. Below is the output from the proxy debug:
[DEBUG ] LazyLoaded interface.utilization_summary
[DEBUG ] LazyLoaded test.ping
[DEBUG ] LazyLoaded mine.update
[DEBUG ] Subprocess Thread-263-Schedule-__proxy_keepalive cleaned up
[DEBUG ] schedule: Job __proxy_keepalive was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG ] schedule: Job __proxy_keepalive was scheduled with a max number of 1
[INFO ] Running scheduled job: __proxy_keepalive with jid 20221101172955648942
[DEBUG ] Subprocess Thread-264-Schedule-__proxy_keepalive added
[DEBUG ] Using importlib_metadata to load entry points
[DEBUG ] LazyLoaded status.proxy_reconnect
[DEBUG ] schedule.handle_func: adding this job to the jobcache with data {'id': 'c73b4f53-2528-4ff9-af51-82837bfd18e2', 'fun': 'status.proxy_reconnect', 'fun_args': [{'proxy_name': 'napalm'}], 'schedule': '__proxy_keepalive', 'jid': '20221101172955648942', 'pid': 11939}
[DEBUG ] schedule.handle_func: Removing /var/cache/salt/proxy/c73b4f53-2528-4ff9-af51-82837bfd18e2/proc/20221101172955648942
[DEBUG ] LazyLoaded interface.utilization_summary
[DEBUG ] LazyLoaded test.ping
[DEBUG ] LazyLoaded mine.update
[DEBUG ] Subprocess Thread-264-Schedule-__proxy_keepalive cleaned up
Check the Event Bus.. There is no event for __schedule_return:
20221101173104365994 {
"_stamp": "2022-11-01T17:31:04.366385",
"minions": [
"c73b4f53-2528-4ff9-af51-82837bfd18e2"
]
}
salt/job/20221101173104365994/new {
"_stamp": "2022-11-01T17:31:04.367102",
"arg": [],
"fun": "schedule.list",
"jid": "20221101173104365994",
"minions": [
"c73b4f53-2528-4ff9-af51-82837bfd18e2"
],
"missing": [],
"tgt": "*",
"tgt_type": "glob",
"user": "root"
}
salt/stats/MWorker-3 {
"_stamp": "2022-11-01T17:31:04.372164",
"stats": {
"publish": {
"mean": 0.006384372711181641,
"runs": 1
}
},
"time": 1014.6155388355255,
"worker": "MWorker-3"
}
salt/job/20221101173104365994/ret/c73b4f53-2528-4ff9-af51-82837bfd18e2 {
"_stamp": "2022-11-01T17:31:04.448417",
"cmd": "_return",
"fun": "schedule.list",
"fun_args": [],
"id": "c73b4f53-2528-4ff9-af51-82837bfd18e2",
"jid": "20221101173104365994",
"retcode": 0,
"return": "schedule:\n enabled: true\n example:\n cron: 0 * * * *\n enabled: true\n function: test.ping\n name: example\n switchport_utilization:\n cron: 0 10,14 * * 1-5\n enabled: true\n function: interface.utilization_summary\n name: switchport_utilization\n splay: 1800\n",
"success": true
}
salt/stats/MWorker-0 {
"_stamp": "2022-11-01T17:31:04.450653",
"stats": {
"_return": {
"mean": 0.002768278121948242,
"runs": 1
}
},
"time": 1012.1096804141998,
"worker": "MWorker-0"
}
Check the Job Cache.. There are no jobs for the schedule:
20221101171415775118:
----------
Arguments:
Function:
schedule.list
StartTime:
2022, Nov 01 17:14:15.775118
Target:
*
Target-type:
glob
User:
root
20221101173104365994:
----------
Arguments:
Function:
schedule.list
StartTime:
2022, Nov 01 17:31:04.365994
Target:
*
Target-type:
glob
User:
root
Expected behavior
Once a scheduled job is executed, the return is published on the event bus and is exposed to the master.
Versions Report
Salt Version:
Salt: 3005
Dependency Versions:
cffi: 1.14.4
cherrypy: 5.6.0
dateutil: 2.8.1
docker-py: 4.4.4
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.0.2
libgit2: Not Installed
M2Crypto: 0.35.2
Mako: Not Installed
msgpack: 0.6.2
msgpack-pure: Not Installed
mysql-python: Not Installed
pycparser: 2.20
pycrypto: 2.6.1
pycryptodome: 3.9.9
pygit2: Not Installed
Python: 3.6.8 (default, Nov 16 2020, 16:55:22)
python-gnupg: Not Installed
PyYAML: 5.3.1
PyZMQ: 18.0.1
smmap: Not Installed
timelib: Not Installed
Tornado: 4.5.3
ZMQ: 4.1.4
System Versions:
dist: centos 7 Core
locale: UTF-8
machine: x86_64
release: 3.10.0-1160.76.1.el7.x86_64
system: Linux
version: CentOS Linux 7 Core
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files or tests are named in the report. Start by tracing proxy minion schedule execution for the shown schedule and the __schedule_return Event Bus path; reproduce the behavior on Salt 3005 and compare it with the expected published return. Done means the scheduled result is published on the Event Bus and exposed to the master.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100