Uncommenting `newline_sequence` in master's config causes `AssertionError`

Overview

• Tested on:
  • 2018.3.3+ds-1 (py2).
  • 2018.3.3+ds-1 (py3).
  • 2019.2.0rc2+ds-1 (py3).
• Uncommenting newline_sequence: '\n' in the master's config causes the following error upon running:
  • AssertionError: newline_sequence set to unknown line ending string.
• Causes the error for both default settings once uncommented:
  • jinja_env:newline_sequence
  • jinja_sls_env:newline_sequence
• Workaround found:
  • Using double-quotes instead of the single-quotes works.
  • I.e. newline_sequence: "\n"

Description of Issue/Question

I've been working on updating the salt-formula for newer master versions. While testing the default blocks under jinja_env and jinja_sls_env, I encountered the error as mentioned above:

    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Rendering Primary Top file failed, render error:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 170, in render_tmpl
    output = render_str(tmplstr, context, tmplpath)
  File "/usr/lib/python2.7/dist-packages/salt/utils/templates.py", line 359, in render_jinja_tmpl
    **env_args)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 305, in __init__
    _environment_sanity_check(self)
  File "/usr/lib/python2.7/dist-packages/jinja2/environment.py", line 99, in _environment_sanity_check
    'newline_sequence set to unknown line ending string.'
AssertionError: newline_sequence set to unknown line ending string.

For further debugging, modified jinja2/environment.py to show the newline_sequence that's being evaluated by the assertion:

AssertionError: newline_sequence set to unknown line ending string: '\\n'

While testing with different values, I found that using double-quotes is a workaround:

# Default Jinja environment options for all templates except sls templates
jinja_env:
  newline_sequence: "\n"

# Jinja environment options for sls templates
jinja_sls_env:
  newline_sequence: "\n"

Testing with spurious values '\a' and "\a" highlights the difference between single and double quotes.

newline_sequence: '\a':

AssertionError: newline_sequence set to unknown line ending string: u'\\a'

newline_sequence: "\a":

AssertionError: newline_sequence set to unknown line ending string: u'\x07'

Setup

Either uncomment the lines in /etc/salt/master directly or provide /etc/salt/master.d/jinja.conf:

# Default Jinja environment options for all templates except sls templates
jinja_env:
  newline_sequence: '\n'

# Jinja environment options for sls templates
jinja_sls_env:
  newline_sequence: '\n'

Any simple .sls that contains Jinja will trigger the assertion error. Such as:

{% set pkg = 'salt-minion' %}

install_pkg:
  pkg.installed:
    - name: {{ pkg }}

Steps to Reproduce Issue

As mentioned above.

Versions Report

Tested with two masters, including upgrading to 2019.2.0rc2+ds-1 on the second master.

2018.3.3+ds-1 (py2):

Salt Version:
           Salt: 2018.3.3
 
Dependency Versions:
           cffi: 0.8.6
       cherrypy: 3.2.2
       dateutil: 1.5
      docker-py: 3.5.0
          gitdb: 0.5.4
      gitpython: 0.3.2 RC1
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: 0.21.1
           Mako: 0.9.1
   msgpack-pure: Not Installed
 msgpack-python: 0.4.6
   mysql-python: 1.2.3
      pycparser: 2.10
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.6 (default, Nov 13 2018, 12:45:42)
   python-gnupg: Not Installed
         PyYAML: 3.10
          PyZMQ: 14.0.1
           RAET: Not Installed
          smmap: 0.8.2
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.0.5
 
System Versions:
           dist: Ubuntu 14.04 trusty
         locale: UTF-8
        machine: x86_64
        release: 3.13.0-66-generic
         system: Linux
        version: Ubuntu 14.04 trusty

2018.3.3+ds-1 (py3):

Salt Version:
           Salt: 2018.3.3

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.7 (default, Oct 22 2018, 11:32:17)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 3.13.0-66-generic
         system: Linux
        version: Ubuntu 18.04 bionic

2019.2.0rc2+ds-1 (py3):

Salt Version:
           Salt: 2019.2.0rc2

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.7 (default, Oct 22 2018, 11:32:17)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: ANSI_X3.4-1968
        machine: x86_64
        release: 3.13.0-66-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Start by reproducing the configuration in /etc/salt/master or /etc/salt/master.d/jinja.conf with the Jinja options shown, then trace rendering through salt/utils/templates.py and the Jinja environment assertion in jinja2/environment.py. Done means the documented single-quoted newline_sequence setting no longer triggers the assertion, with regression coverage for both jinja_env and jinja_sls_env.