saltstack / saltstack/salt

grains_refresh_every directive not reliably creating schedule

Open
#56,003 2 comments 0 reactions 1 assignee View on GitHub

@xeacott is already working on this.

Since Feb 7, 2020.

bug Windows
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

Setting grains_refresh_every does not reliably create a schedule in _schedule.conf. Running the minion in debug shows that the directive is processed, but immediately appears to get deleted.

Relevant debug log

C:\salt>salt-minion-debug
The salt-minion service is not started.

More help is available by typing NET HELPMSG 3521.

[DEBUG ] Reading configuration from c:\salt\conf\minion
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d_schedule.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d_schedule.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\network.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\network.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\rainhail_minion_win.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\rainhail_minion_win.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\s3.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\s3.conf
[DEBUG ] Multiprocessing queue logging configured for the process running under PID: 8492 at log level 10
[DEBUG ] Marking 'is_ip' as a jinja filter
[DEBUG ] Marking 'is_ipv4' as a jinja filter
[DEBUG ] Marking 'is_ipv6' as a jinja filter
[DEBUG ] Marking 'ipv4' as a jinja filter
[DEBUG ] Marking 'ipv6' as a jinja filter
[DEBUG ] Marking 'ipaddr' as a jinja filter
[DEBUG ] Marking 'ip_host' as a jinja filter
[DEBUG ] Marking 'network_hosts' as a jinja filter
[DEBUG ] Marking 'network_size' as a jinja filter
[DEBUG ] Marking 'gen_mac' as a jinja filter
[DEBUG ] Marking 'mac_str_to_bytes' as a jinja filter
[DEBUG ] Marking 'dns_check' as a jinja filter
[DEBUG ] Multiprocessing queue logging configured for the process running under PID: 8492 at log level 10
[DEBUG ] Configuration file path: c:\salt\conf\minion
[WARNING ] Insecure logging configuration detected! Sensitive data may be logged.
[INFO ] Setting up the Salt Minion "demotom038.rainhail.com"
[DEBUG ] Marking 'base64_encode' as a jinja filter
[DEBUG ] Marking 'base64_decode' as a jinja filter
[DEBUG ] Marking 'md5' as a jinja filter
[DEBUG ] Marking 'sha256' as a jinja filter
[DEBUG ] Marking 'sha512' as a jinja filter
[DEBUG ] Marking 'hmac' as a jinja filter
[DEBUG ] Marking 'random_hash' as a jinja filter
[DEBUG ] Marking 'rand_str' as a jinja filter
[DEBUG ] Marking 'file_hashsum' as a jinja filter
[DEBUG ] Marking 'http_query' as a jinja filter
[DEBUG ] Marking 'strftime' as a jinja filter
[DEBUG ] Marking 'date_format' as a jinja filter
[DEBUG ] Marking 'yaml_dquote' as a jinja filter
[DEBUG ] Marking 'yaml_squote' as a jinja filter
[DEBUG ] Marking 'yaml_encode' as a jinja filter
[DEBUG ] Marking 'raise' as a jinja global
[DEBUG ] Marking 'match' as a jinja test
[DEBUG ] Marking 'equalto' as a jinja test
[DEBUG ] Marking 'skip' as a jinja filter
[DEBUG ] Marking 'sequence' as a jinja filter
[DEBUG ] Marking 'to_bool' as a jinja filter
[DEBUG ] Marking 'tojson' as a jinja filter
[DEBUG ] Marking 'quote' as a jinja filter
[DEBUG ] Marking 'regex_escape' as a jinja filter
[DEBUG ] Marking 'regex_search' as a jinja filter
[DEBUG ] Marking 'regex_match' as a jinja filter
[DEBUG ] Marking 'regex_replace' as a jinja filter
[DEBUG ] Marking 'uuid' as a jinja filter
[DEBUG ] Marking 'unique' as a jinja filter
[DEBUG ] Marking 'min' as a jinja filter
[DEBUG ] Marking 'max' as a jinja filter
[DEBUG ] Marking 'avg' as a jinja filter
[DEBUG ] Marking 'union' as a jinja filter
[DEBUG ] Marking 'intersect' as a jinja filter
[DEBUG ] Marking 'difference' as a jinja filter
[DEBUG ] Marking 'symmetric_difference' as a jinja filter
[DEBUG ] Created pidfile: c:\salt\var\run\salt-minion.pid
[INFO ] Starting up the Salt Minion
[DEBUG ] AsyncEventPublisher PUB socket URI: 4510
[DEBUG ] AsyncEventPublisher PULL socket URI: 4511
[INFO ] Starting pull socket on 4511
[DEBUG ] SaltEvent PUB socket URI: 4510
[DEBUG ] SaltEvent PULL socket URI: 4511
[DEBUG ] Initializing new IPCClient for path: 4510
[DEBUG ] Grains cache last modified 406 seconds ago and cache expiration is set to 300. Grains cache expired. Refreshing.
[DEBUG ] Reading configuration from c:\salt\conf\minion
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d_schedule.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d_schedule.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\network.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\network.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\rainhail_minion_win.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\rainhail_minion_win.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\s3.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\s3.conf
[INFO ] Retrieved install_date from file
[DEBUG ] Reading configuration from c:\salt\conf\minion
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d_schedule.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d_schedule.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\network.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\network.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\rainhail_minion_win.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\rainhail_minion_win.conf
[DEBUG ] Including configuration from 'c:\salt\conf\minion.d\s3.conf'
[DEBUG ] Reading configuration from c:\salt\conf\minion.d\s3.conf
[DEBUG ] Requesting URL http://webadmin.rainhail.com/public/all_tomcat_hosts.php using POST method
[DEBUG ] Using backend: tornado
[DEBUG ] Response Status Code: 200
[INFO ] Creating minion process manager
[DEBUG ] Process Manager starting!
[DEBUG ] Process Manager starting!
[DEBUG ] Connecting to master. Attempt 1 (infinite attempts)
[DEBUG ] Master URI: tcp://172.24.55.45:4506
[DEBUG ] Popen(['git', 'version'], cwd=C:\salt, universal_newlines=False, shell=None)
[DEBUG ] Initializing new AsyncAuth for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506')
[DEBUG ] Generated random reconnect delay between '1000ms' and '11000ms' (4292)
[DEBUG ] Setting zmq_reconnect_ivl to '4292ms'
[DEBUG ] Setting zmq_reconnect_ivl_max to '11000ms'
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506', 'clear')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://172.24.55.45:4506
[DEBUG ] Trying to connect to: tcp://172.24.55.45:4506
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] Decrypting the current master AES key
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG ] Loaded minion key: c:\salt\conf\pki\minion\minion.pem
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] Connecting the Minion to the Master publish port, using the URI: tcp://172.24.55.45:4505
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: c:\salt\conf\pki\minion\minion.pem
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506', 'aes')
[DEBUG ] Re-using AsyncAuth for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://172.24.55.45:4506
[DEBUG ] Trying to connect to: tcp://172.24.55.45:4506
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] Loaded minion key: c:\salt\conf\pki\minion\minion.pem
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] Retrieving grains from cache
[DEBUG ] Initializing new Schedule
[DEBUG ] LazyLoaded platform.is_windows
[DEBUG ] LazyLoaded path.which
[DEBUG ] LazyLoaded timezone.get_offset
[DEBUG ] LazyLoaded cmd.run_all
[INFO ] Executing command ['tzutil', '/g'] in directory 'C:\Users\heiselmanjx'
[DEBUG ] stdout: Central Standard Time
[DEBUG ] LazyLoaded config.merge
[DEBUG ] SaltEvent PUB socket URI: 4510
[DEBUG ] SaltEvent PULL socket URI: 4511
[DEBUG ] Initializing new IPCClient for path: 4511
[DEBUG ] Sending event: tag = /salt/minion/minion_schedule_delete_complete; data = {'schedule': {'__mine_interval': {'maxrunning': 2, 'function': 'mine.update', 'minutes': 60, 'jid_include': True, 'enabled': True, 'return_job': False, 'run_on_start': True}}, 'complete': True, '_stamp': '2020-01-28T14:34:51.763513'}
[DEBUG ] Persisting schedule
[DEBUG ] Closing IPCMessageClient instance
[DEBUG ] LazyLoaded mine.update
[INFO ] Updating job settings for scheduled job: __mine_interval
[DEBUG ] SaltEvent PUB socket URI: 4510
[DEBUG ] SaltEvent PULL socket URI: 4511
[DEBUG ] Initializing new IPCClient for path: 4511
[DEBUG ] Sending event: tag = /salt/minion/minion_schedule_add_complete; data = {'schedule': {'__mine_interval': {'maxrunning': 2, 'function': 'mine.update', 'enabled': True, 'jid_include': True, 'minutes': 60, 'return_job': False, 'run_on_start': True}}, 'complete': True, '_stamp': '2020-01-28T14:34:51.763513'}
[DEBUG ] Persisting schedule
[DEBUG ] Closing IPCMessageClient instance
[INFO ] Added mine.update to scheduler
[DEBUG ] SaltEvent PUB socket URI: 4510
[DEBUG ] SaltEvent PULL socket URI: 4511
[DEBUG ] Initializing new IPCClient for path: 4511
[DEBUG ] Sending event: tag = /salt/minion/minion_schedule_delete_complete; data = {'schedule': {'__mine_interval': {'maxrunning': 2, 'function': 'mine.update', 'enabled': True, 'jid_include': True, 'minutes': 60, 'return_job': False, 'run_on_start': True}}, 'complete': True, '_stamp': '2020-01-28T14:34:51.763513'}
[DEBUG ] Persisting schedule
[DEBUG ] Closing IPCMessageClient instance
[DEBUG ] SaltEvent PUB socket URI: 4510
[DEBUG ] SaltEvent PULL socket URI: 4511
[DEBUG ] Initializing new IPCClient for path: 4511
[DEBUG ] Sending event: tag = /salt/minion/minion_schedule_delete_complete; data = {'schedule': {'__mine_interval': {'maxrunning': 2, 'function': 'mine.update', 'enabled': True, 'jid_include': True, 'minutes': 60, 'return_job': False, 'run_on_start': True}}, 'complete': True, '_stamp': '2020-01-28T14:34:51.763513'}
[DEBUG ] Persisting schedule
[DEBUG ] Closing IPCMessageClient instance
[INFO ] Minion is starting as user 'RAINHAIL\heiselmanjx'
[DEBUG ] Minion 'demotom038.rainhail.com' trying to tune in
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506', 'aes')
[DEBUG ] Initializing new AsyncAuth for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://172.24.55.45:4506
[DEBUG ] Trying to connect to: tcp://172.24.55.45:4506
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506', 'aes')
[DEBUG ] Initializing new AsyncAuth for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://172.24.55.45:4506
[DEBUG ] Trying to connect to: tcp://172.24.55.45:4506
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[INFO ] Minion is ready to receive requests!
[DEBUG ] Enabling the grains refresher. Will run every 720 minutes.
[DEBUG ] Minion of 'webadminubuntu02.rainhail.com' is handling event tag '/salt/minion/minion_schedule_delete_complete'
[DEBUG ] Minion of 'webadminubuntu02.rainhail.com' is handling event tag '/salt/minion/minion_schedule_add_complete'
[DEBUG ] Minion of 'webadminubuntu02.rainhail.com' is handling event tag '/salt/minion/minion_schedule_delete_complete'
[DEBUG ] Minion of 'webadminubuntu02.rainhail.com' is handling event tag '/salt/minion/minion_schedule_delete_complete'
[DEBUG ] schedule: Job __mine_interval was scheduled with jid_include, adding to cache (jid_include defaults to True)
[DEBUG ] schedule: Job __mine_interval was scheduled with a max number of 2
[INFO ] Running scheduled job: __mine_interval
[DEBUG ] Multiprocessing queue logging configured for the process running under PID: 6000 at log level 10
[DEBUG ] schedule.handle_func: adding this job to the jobcache with data {'schedule': '__mine_interval', 'fun_args': [], 'fun': 'mine.update', 'pid': 6000, 'id': 'demotom038.rainhail.com', 'jid': '20200128083454372907'}
[DEBUG ] LazyLoaded mine.update
[DEBUG ] LazyLoaded config.merge
[DEBUG ] schedule.handle_func: Removing c:\salt\var\cache\salt\minion\proc\20200128083454372907
[DEBUG ] Popen(['git', 'version'], cwd=C:\salt, universal_newlines=False, shell=None)
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506', 'aes')
[DEBUG ] Initializing new AsyncAuth for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://172.24.55.45:4506
[DEBUG ] Trying to connect to: tcp://172.24.55.45:4506
[DEBUG ] Initializing new AsyncZeroMQReqChannel for ('c:\salt\conf\pki\minion', 'demotom038.rainhail.com', 'tcp://172.24.55.45:4506', 'clear')
[DEBUG ] Connecting the Minion to the Master URI (for the return server): tcp://172.24.55.45:4506
[DEBUG ] Trying to connect to: tcp://172.24.55.45:4506
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] Decrypting the current master AES key
[DEBUG ] salt.crypt.get_rsa_key: Loading private key
[DEBUG ] salt.crypt._get_key_with_evict: Loading private key
[DEBUG ] Loaded minion key: c:\salt\conf\pki\minion\minion.pem
[DEBUG ] salt.crypt.get_rsa_pub_key: Loading public key
[DEBUG ] Closing AsyncZeroMQReqChannel instance
[DEBUG ] Closing AsyncZeroMQReqChannel instance
The Salt Minion is shutdown. Minion received a SIGINT. Exited.
Terminate batch job (Y/N)? y

Setup

In /etc/salt/minion or C:\salt\conf\minion:
grains_refresh_every: 720

Steps to Reproduce Issue

Observe that the _schedule.conf file doesn't get updated with the __update_grains schedule item

Versions Report

Salt Version:
Salt: 2019.2.0

Dependency Versions:
cffi: 1.11.5
cherrypy: 17.4.1
dateutil: 2.7.5
docker-py: Not Installed
gitdb: 2.0.5
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.10
libgit2: Not Installed
libnacl: 1.6.1
M2Crypto: Not Installed
Mako: 1.0.7
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.19
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: Not Installed
Python: 3.5.4 (v3.5.4:3f56838, Aug 8 2017, 02:17:05) [MSC v.1900 64 bit (AMD64)]
python-gnupg: 0.4.3
PyYAML: 3.13
PyZMQ: 17.1.2
RAET: Not Installed
smmap: 2.0.5
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.2.5

System Versions:
dist:
locale: cp1252
machine: AMD64
release: 2019Server
system: Windows
version: 2019Server 10.0.17763 SP0 Multiprocessor Free

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.