YAML refs aren't dereferenced in local Pillars

Description of Issue/Question

When using YAML refs/anchors in Pillar SLS of a local/masterless setup, refs aren't de-referenced before passing them into other contexts which leads to some quite unexpected results e.g. when processing those Pillar data further in a Jinja context, where updating a value within the data suddenly changes other data as well, as they all share the same referenced object.

Setup

State Template: bug.sls:

{% load_yaml as yaml_input %}
anchors:
    project:
        repositories: &id001
            files:
            -   name: salt-top-states
projects:
    a:
        repositories: *id001
    b:
        repositories: *id001
    c:
        repositories:
            files:
            -   name: foobar
    d:
        repositories: *id001
{% endload %}

{% set projects = salt['pillar.get']('projects', {}) %}
{# set projects = yaml_input['projects'] #}

{% for project, projectdata in projects.items() %}
  {% do projectdata.update({
    'config': [],
  }) %}
  {% for repo in projectdata.repositories.files %}
    {% do repo.update({
      "path": project,
    }) %}
  {% endfor %}
{% endfor %}

{% for project, projectdata in projects.items() %}
  {% for repo in projectdata.repositories.files %}
    {% do projectdata.config.append(project) %}
    {% do projectdata.config.append(repo.path) %}
  {% endfor %}
{% endfor %}

{%- for project, projectdata in projects.items() %}
{{ project }}:
    - {{ projectdata.config|yaml }}
{%- endfor %}

Steps to Reproduce Issue

When using bug.sls provided above with the line {% set projects = yaml_input['projects'] %} not commented out ({# ... #}), the output generated by salt-call slsutil.renderer salt://bug.sls looks correct/as expected:

local:
    ----------
    a:
        |_
          - a
          - a
    b:
        |_
          - b
          - b
    c:
        |_
          - c
          - c
    d:
        |_
          - d
          - d

When using the inlined YAML as Pillar SLS instead and commenting out the line {% set projects = yaml_input['projects'] %} again, the result looks like this (please note the non-matching elements (d in a, d in b):

local:
    ----------
    a:
        |_
          - a
          - d
    b:
        |_
          - b
          - d
    c:
        |_
          - c
          - c
    d:
        |_
          - d
          - d

This indicates that YAML refs are simply passed through as Python references up to the point where Jinja processes the data of the dictionary, so instead of working on loop-scoped data in Jinja, one actually processes referenced data crossing loop-scope boundaries.

This can be reproduced using this Python-only code-snippet:

#!/usr/bin/env python3

import pprint

from copy import deepcopy

from salt.utils.jinja import SerializerExtension
from salt.renderers.yaml import render as yaml_render

indoc = """
anchors:
    project:
        repositories: &id001
            files:
            -   name: salt-top-states
projects:
    a:
        repositories: *id001
    b:
        repositories: *id001
    c:
        repositories:
            files:
            -   name: foobar
    d:
        repositories: *id001
"""

def domagic(data, name=None):
  for project, projectdata in data.items():
    projectdata.update({
      'config': [],
    })
    for repo in projectdata['repositories']['files']:
      repo.update({
        "path": project,
      })

  for project, projectdata in data.items():
    for repo in projectdata['repositories']['files']:
      projectdata['config'].append(project)
      projectdata['config'].append(repo['path'])

  print("{}, {}".format(name, type(data)))
  for project, projectdata in data.items():
    print(project)
    pprint.pprint(projectdata['config'])

via_yaml_load = SerializerExtension.load_yaml(None, indoc)
via_yaml_render = yaml_render(indoc)

domagic(via_yaml_load['projects'], 'yaml_load')
domagic(via_yaml_render['projects'], 'yaml_render')
domagic(deepcopy(via_yaml_render['projects']), 'yaml_render+deepcopy')

Output:

yaml_load, <class 'dict'>
a
['a', 'a']
b
['b', 'b']
c
['c', 'c']
d
['d', 'd']
yaml_render, <class 'salt.utils.odict.OrderedDict'>
a
['a', 'd']
b
['b', 'd']
c
['c', 'c']
d
['d', 'd']
yaml_render+deepcopy, <class 'salt.utils.odict.OrderedDict'>
a
['a', 'd']
b
['b', 'd']
c
['c', 'c']
d
['d', 'd']

The output shows a few things:

• SerializerExtension.load_yaml from salt.utils.jinja provides a regular dictionary and works as expected
• render from salt.renderers.yaml doesn't work as expected and provides an OrderedDictionary (via salt.utils.odict).
• trying to de-reference the OrderedDictionary using copy.deepcopy doesn't work, but I'm not sure whether that's a general problem of OrderedDictionary or due to the custom Py27/Py3 compatibility layer in salt.utils.odict

In the end, I want to be able to simply not having to care about implementation internals when processing YAML/Pillar data in a Renderer/Jinja context - I expect Pillar data to be fully de-referenced when working with them.

Versions Report

Also reproduced in 2018.3.3.

Salt Version:
           Salt: 2019.2.0

Dependency Versions:
           cffi: Not Installed
       cherrypy: Not Installed
       dateutil: 2.6.1
      docker-py: Not Installed
          gitdb: 2.0.3
      gitpython: 2.1.8
          ioflo: Not Installed
         Jinja2: 2.10
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: Not Installed
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 3.6.5 (default, Apr  1 2018, 05:46:30)
   python-gnupg: 0.4.1
         PyYAML: 3.12
          PyZMQ: 16.0.2
           RAET: Not Installed
          smmap: 2.0.3
        timelib: Not Installed
        Tornado: 4.5.3
            ZMQ: 4.2.5

System Versions:
           dist: Ubuntu 18.04 bionic
         locale: UTF-8
        machine: x86_64
        release: 4.15.0-48-generic
         system: Linux
        version: Ubuntu 18.04 bionic

Start with salt.renderers.yaml.render and compare its handling of YAML anchors with salt.utils.jinja.SerializerExtension.load_yaml, using the Python-only reproduction in the issue. Trace how the OrderedDict reaches Pillar and Jinja processing, then verify that aliased values are independent and that the reported a/b/d output no longer shares mutations.