saltstack / saltstack/salt

[BUG] Salt exit with bound method Client.__del__ of <etcd.client.Client object at ignored

Open
#57,196 24 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Confirmed Core severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
After upgrading salt master and minion to 2019.2.4, we started getting an error
We have upgraded due to CVE issue

ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored

This error occurs after several execution of salt run.
When we restart the salt-master then it goes for a while, but after few hours it starts appearing again

Setup
(Please provide relevant configs and/or SLS files (be sure to remove sensitive info).

Steps to Reproduce the behavior
(Include debug logs if possible and relevant)

salt 'target' cmd.run 'ls -l /var' -l debug

[DEBUG   ] .Passing on saltutil error. Key 'u'retcode' missing from client return. This may be an error in the client
[DEBUG   ] return event: {'example1.com': {u'failed': True}}
[DEBUG   ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO    ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG   ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions HTTP/1.1" 200 None
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None
[DEBUG   ] Could not LazyLoad etcd.init_kwargs: 'etcd.init_kwargs' is not available.
[INFO    ] etcd: Setting up client with params: {u'username': u'saltmaster', u'protocol': u'http', u'ca_cert': u'None', u'allow_redirect': False, u'srv_domain': None, u'host': u'192.168.1.98', u'read_timeout': 60, u'password': u'test_pass', u'port': 2379, u'allow_reconnect': True, u'cert': u'None'}
[DEBUG   ] Starting new HTTP connection (1): 192.168.1.98:2379
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/machines HTTP/1.1" 200 23
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache HTTP/1.1" 200 178
[DEBUG   ] http://192.168.1.98:2379 "GET /v2/keys/salt_cache/minions?recursive=true HTTP/1.1" 200 None

[DEBUG   ] Closing IPCMessageSubscriber instance
ERROR: Minions returned with non-zero exit code
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method Client.__del__ of <etcd.client.Client object at 0x7f4f62781810>> ignored

Warning on minion log:

2020-05-05 17:00:15,248 [salt.utils.parsers:1071][WARNING ][1844] Minion received a SIGTERM. Exiting.
2020-05-05 17:02:00,540 [salt.utils.parsers:1071][WARNING ][25113] Minion received a SIGTERM. Exiting.
2020-05-06 01:07:57,521 [salt.utils.parsers:1071][WARNING ][25358] Minion received a SIGTERM. Exiting.

For simple salt.run it occurs after 10-15 continuous salt call from master

Expected behavior
Error must go, instead it should print expected output

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

salt --versions-report (Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)
salt --versions-report
Salt Version:
           Salt: 2019.2.4

Dependency Versions:
           cffi: 1.6.0
       cherrypy: Not Installed
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: 0.26.4
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: 3.9.7
         pygit2: 0.26.4
         Python: 2.7.5 (default, Jun 11 2019, 14:33:56)
   python-gnupg: 0.4.4
         PyYAML: 3.10
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4

System Versions:
           dist: redhat 7.6 Maipo
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-1127.el7.x86_64
         system: Linux
        version: Red Hat Enterprise Linux Server 7.6

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reported salt 'target' cmd.run 'ls -l /var' -l debug workflow against the stated Salt 2019.2.4 and etcd setup, then compare the master and minion logs with the versions report. Done means repeated calls no longer produce the Client.__del__ TypeError and return the expected command output without the related non-zero exit.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
databases, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.