saltstack / saltstack/salt

[BUG] cannot get full output using Salt CLi net.cli function napalm_network module is large

Open
#59,167 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug pending-discussion
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description
When running commands on Salt CLI like net.cli function from NAPALM_NETWORK MODULE of Salt ,If the output/response of the command is large it simply trims the output completely.
In other words Salt is not able to to fetch switch’s routing table, other than a very small VRF table with 100 - 200 routes. If at all the data is managed to be received with 100 - 200, it is taking more than 5 mins. Otherwise, it simply trims the data completely and display "VALUE_TRIMMED" as a result returned.
And trying to run the same command 2 or more time will result in
""
Minion did not return. [Not connected]
"""
I have tried changing the the master and minions configuration with max_event_size to 5242880 and kept increasing this to check if this would work. But no luck in getting the full output for the command
Also tried passing the flag to increase the timing to wait for the response from the minion using --timeout flag, This also doesn't seem to help.

Whereas, When using a python script or calling pyeapi I can get the complete output of the command with in 2 -3 seconds.
For example:
from napalm import get_network_driver
driver = get_network_driver('eos')
device = driver(hostname='arista-sw-d2', usename='username', password='password')
device.open()
device.cli(['show ip route vrf all'])
---output snipped ----

Setup
VM1 configured as salt-master

salt-master service running on Ubuntu 16.04
salt-master version 3001.1
memory 48 GB
cpus 20

VM2 configured as salt-minion for running the proxy services for network devices using napalm

salt-minion service running on Ubuntu 16.04
salt-minion version 3001.1
memory 64 GB
cpus 24
napalm version 2.5.0

https://docs.saltstack.com/en/3000/ref/proxy/all/salt.proxy.napalm.html#module-salt.proxy.napalm

Steps to Reproduce the behavior
tried many ways of executing the command show ip route vrf all from Salt on the device.
Using napalm.call

root@dc4-net-salt-prd-01:~# salt arista-sw-d2 napalm.call cli commands=["show ip route vrf all"]
arista-sw-d2:
    ----------
    comment:
    out:
        ----------
        show ip route vrf all:
            VALUE_TRIMMED
    result:
        True
Using pyeapi
salt arista-sw-d2.dc4-in.nvidia.com pyeapi.run_commands "show ip route vrf all" host='arista-sw-d2' username='username' password='********' transport='https' -t 300
arista-sw-d2:
    ----------
    comment:
    out:
        ----------
        show ip route vrf all:
            VALUE_TRIMMED
    result:
        True
Using net.cli
salt arista-sw-d2 net.cli "show ip route vrf all" -t 300
arista-sw-d2:
    ----------
    comment:
    out:
        ----------
        show ip route vrf all:
            VALUE_TRIMMED
    result:
        True

Expected behavior
Running command "show ip route vrf all" on a network device using net.cli should display the complete output, not the "VALUE_TRIMMED"

Screenshots
Added screenshots to help explain the problem.
image

Versions Report

salt --versions-report
root:~# salt --versions-report
Salt Version:
           Salt: 3001.1

Dependency Versions:
           cffi: 1.14.0
       cherrypy: 3.2.3
       dateutil: 2.4.2
      docker-py: Not Installed
          gitdb: 0.6.4
      gitpython: 1.0.1
         Jinja2: 2.11.2
        libgit2: 0.24.0
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.6.2
   mysql-python: Not Installed
      pycparser: 2.20
       pycrypto: 2.6.1
   pycryptodome: 3.4.7
         pygit2: 0.24.0
         Python: 3.5.2 (default, Oct  7 2020, 17:19:02)
   python-gnupg: 0.3.8
         PyYAML: 5.3.1
          PyZMQ: 19.0.1
          smmap: 0.9.0
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.2

System Versions:
           dist: ubuntu 16.04 Xenial Xerus
         locale: UTF-8
        machine: x86_64
        release: 4.4.0-193-generic
         system: Linux
        version: Ubuntu 16.04 Xenial Xerus

There is no differences in master/minion versions.
(Provided by running salt --versions-report.)

Additional context
Add any other context about the problem here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the net.cli and napalm.call entry points and the salt.proxy.napalm documentation linked in the report. Reproduce show ip route vrf all with the provided Salt and NAPALM versions, then trace where the response becomes VALUE_TRIMMED and how max_event_size and timeout are handled. Done means the complete command output is returned without the minion becoming disconnected.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.