[BUG] Salt exit with bound method Client.__del__ of <etcd.client.Client object at ignored
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 5.6k
- Avg merge
- 2d 44m
- Merged PRs (30d)
- 80
Description
Description
After upgrading salt master and minion to 2019.2.4, we started getting an error
We have upgraded due to CVE issue
ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored
This error occurs after several execution of salt run.
When we restart the salt-master then it goes for a while, but after few hours it starts appearing again
Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info).
Steps to Reproduce the behavior
(Include debug logs if possible and relevant)
salt 'target' cmd.run 'ls -l /var' -l debug
[DEBUG ] .Passing on saltutil error. Key 'u'retcode' missing from client return. This may be an error in the client
[DEBUG ] return event: {'example1.com': {u'failed': True}}
[DEBUG ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions HTTP/1.1" 200 None
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None
[DEBUG ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None
[DEBUG ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored
Warning on minion log:
2020-05-05 17:00:15,248 [salt.utils.parsers:1071][WARNING ][1844] Minion received a SIGTERM. Exiting.
2020-05-05 17:02:00,540 [salt.utils.parsers:1071][WARNING ][25113] Minion received a SIGTERM. Exiting.
2020-05-06 01:07:57,521 [salt.utils.parsers:1071][WARNING ][25358] Minion received a SIGTERM. Exiting.
For simple salt.run it occurs after 10-15 continuous salt call from master
Expected behavior
Error must go, instead it should print expected output
Screenshots
If applicable, add screenshots to help explain your problem.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)salt --versions-report
Salt Version:
Salt: 2019.2.4
Dependency Versions:
cffi: 1.6.0
cherrypy: Not Installed
dateutil: 1.5
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.7.2
libgit2: 0.26.4
libnacl: Not Installed
M2Crypto: 0.21.1
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.14
pycrypto: 2.6.1
pycryptodome: 3.9.7
pygit2: 0.26.4
Python: 2.7.5 (default, Jun 11 2019, 14:33:56)
python-gnupg: 0.4.4
PyYAML: 3.10
PyZMQ: 15.3.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 4.2.1
ZMQ: 4.1.4
System Versions:
dist: redhat 7.6 Maipo
locale: UTF-8
machine: x86_64
release: 3.10.0-1127.el7.x86_64
system: Linux
version: Red Hat Enterprise Linux Server 7.6
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by running the reported salt 'target' cmd.run 'ls -l /var' -l debug workflow against the stated Salt 2019.2.4 and etcd setup, then compare the master and minion logs with the versions report. Done means repeated calls no longer produce the Client.__del__ TypeError and return the expected command output without the related non-zero exit.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- databases, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100