saltstack / saltstack/salt

Timeout value problem with napalm.junos_cli and sometimes RPC

Open
#58,108 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug severity-medium
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

When I run napalm.junos_cli commands, they have issues with a timeout not getting set. There are a couple variants of this and I'm not 100% sure they're related, but they look awfully close.

issue the first

It looks like there should be a timeout passed in for a cli command, and there isn't. I have tried setting -t <val> and --timeout=<val> both immediately after $RTR and at the endo of the command line and there's no difference.

salt@9bcd18b7855c:~$ salt $RTR napalm.junos_cli "show version"
cfr1.ix1.dfw:
    The minion function caused an exception: Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/jnpr/junos/device.py", line 206, in timeout
        self._conn.timeout = int(value)
    TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/lib/python3/dist-packages/salt/metaproxy/proxy.py", line 480, in thread_return
        opts, data, func, args, kwargs
      File "/usr/lib/python3/dist-packages/salt/executors/direct_call.py", line 12, in execute
        return func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/utils/napalm.py", line 515, in func_wrapper
        ret = func(*args, **kwargs)
      File "/usr/lib/python3/dist-packages/salt/modules/napalm_mod.py", line 930, in junos_cli
        command, format=format, dev_timeout=dev_timeout, dest=dest, **kwargs
      File "/usr/lib/python3/dist-packages/salt/modules/junos.py", line 174, in wrapper
        conn.timeout = kwargs.pop("dev_timeout", None)
      File "/usr/local/lib/python3.6/dist-packages/jnpr/junos/device.py", line 209, in timeout
        "could not convert timeout value of %s to an " "integer" % (value)
    RuntimeError: could not convert timeout value of None to an integer
ERROR: Minions returned with non-zero exit code
issue the second

Although some direct `napalm.junos_rpc' commands work, like this one:

salt@9bcd18b7855c:~$ salt $RTR napalm.junos_rpc get-system-information
cfr1.ix1.dfw:
    ----------
    comment:
    out:
        ----------
        system-information:
            ----------
            hardware-model:
                mx10003
            host-name:
                cfr1.ix1.dfw
...

some of the more involved ones do not, and they have what looks like a similar timeout issue.
If I run the command salt $RTR net.cli "request vmhost snapshot" -t 120
the -t 120 timeout gets obeyed but there is an rpc timeout error:

salt@9bcd18b7855c:~$ salt $RTR net.cli "request vmhost snapshot" -t 120
cfr1.ix1.dfw:
    ----------
    comment:
    out:
        ----------
        request vmhost snapshot:
            invalid command: request vmhost snapshot: RpcTimeoutError(host: cfr1.ix1.dfw, cmd: command, timeout: 60)
    result:
        True

I have tried any number of --dev_timeout <val> and --dev_timeout=<val> and dev_timeout= parameters , at various places in the command line, and nothing I do makes this go away. Setting --dev-timeout to some value >60 always leaves me with the same error message, always displaying "timeout: 60". So whatever I'm trying to pass in isn't getting relayed down. One example:

salt@9bcd18b7855c:~$ time salt $RTR net.cli -t 120 "request vmhost snapshot" dev_timeout=5000
cfr1.ix1.dfw:
    ----------
    comment:
    out:
        ----------
        request vmhost snapshot:
            invalid command: request vmhost snapshot: RpcTimeoutError(host: cfr1.ix1.dfw, cmd: command, timeout: 60)
    result:
        True

real	1m1.384s
user	0m1.740s
sys	0m0.234s
Setup

I am running salt-master in one docker container and salt-proxy in another. All version 3001, freshly built.

Steps to Reproduce Issue

See above. I'm happy to run more tests, this is easy to reproduce.

Versions Report

Master and minion are build from the same container base:

salt@9bcd18b7855c:~$ salt --versions-report
Salt Version:
           Salt: 3001

Dependency Versions:
           cffi: 1.14.1
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
         Jinja2: 2.11.2
        libgit2: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: 3.4.7
         pygit2: Not Installed
         Python: 3.6.9 (default, Jul 17 2020, 12:50:27)
   python-gnupg: 0.4.1
         PyYAML: 5.3.1
          PyZMQ: 17.1.2
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: ubuntu 18.04 Bionic Beaver
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 5.3.0-1030-gcp
         system: Linux
        version: Ubuntu 18.04 Bionic Beaver

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the commands in the issue, then trace timeout handling through salt/modules/napalm_mod.py, salt/modules/junos.py, and salt/utils/napalm.py, starting at napalm.junos_cli and net.cli. Done means the supplied timeout reaches the Junos CLI and RPC operations instead of becoming None or remaining fixed at 60 seconds.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.