saltstack / saltstack/salt

2019.2 External file_tree pillar is broken with salt-ssh

Open
#53,620 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug Salt-SSH severity-high
Dominant language
Python
Stars
15.7k
Forks
5.6k
Avg merge
2d 44m
Merged PRs (30d)
80

Description

Description of Issue

Having configured a file tree external pillar (without templating enabled), salt-ssh call results in the following trace:

[ERROR   ] TypeError encountered executing state.apply: Object of type bytes is not JSON serializable
Traceback (most recent call last):
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/client/ssh/__init__.py", line 1159, in run_wfunc
    result = self.wfuncs[self.fun](*self.args, **self.kwargs)
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/client/ssh/wrapper/state.py", line 513, in apply_
    return highstate(**kwargs)
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/client/ssh/wrapper/state.py", line 705, in highstate
    roster_grains)
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/client/ssh/state.py", line 193, in prep_trans_tar
    salt.utils.json.dump(pillar, fp_)
  File "/Users/user1/Library/Python/3.7/lib/python/site-packages/salt/utils/json.py", line 121, in dump
    return json_module.dump(obj, fp, **kwargs)  # future lint: blacklisted-function
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/__init__.py", line 179, in dump
    for chunk in iterable:
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 431, in _iterencode
    yield from _iterencode_dict(o, _current_indent_level)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 405, in _iterencode_dict
    yield from chunks
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 438, in _iterencode
    o = _default(o)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/json/encoder.py", line 179, in default
    raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type bytes is not JSON serializable

This does not occur when using Python2.7.

Setup

Setup pillar.file_tree

ext_pillar:
  - file_tree:
        root_dir: file-tree
        follow_dir_links: False
        keep_newline: True

Place binary file in file_tree.

Steps to Reproduce Issue

Attempt to apply state using salt-ssh.

Versions Report
Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.7.3
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.10.1
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: 1.0.12
   msgpack-pure: Not Installed
 msgpack-python: 0.6.1
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.7.3 (default, Mar 27 2019, 09:23:15)
   python-gnupg: Not Installed
         PyYAML: 3.13
          PyZMQ: 18.0.1
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.3.1

System Versions:
           dist:
         locale: UTF-8
        machine: x86_64
        release: 18.6.0
         system: Darwin
        version: 10.14.5 x86_64

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in salt/client/ssh/state.py at prep_trans_tar, where the pillar is written with salt.utils.json.dump, and review salt/utils/json.py. Reproduce the issue with a file_tree pillar containing a binary file and a salt-ssh state.apply call under Python 3. Done means the pillar can be transferred without the bytes-not-JSON-serializable error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.