Core napalm grains assigned after custom grains

Description of Issue

According to https://docs.saltstack.com/en/latest/topics/grains/#precedence core grains are assigned first and thus should be overwritten by custom grains in _grains/. However, this is not the case for core grains included in salt/grains/napalm.py

Setup

Need an actual or virtual device capable of being controlled by napalm proxy minion. Need to then configure proxy nominally via pillar data.

Steps to Reproduce Issue

Create custom grain 'interfaces' to override salt-napalm included 'interfaces'. Does not work as described in link above

Test to demonstrate problem (some output modified / redacted)
Created custom grain in issue.py

# /srv/salt/_grains/issue.py

import logging

__proxyenabled__ = ['napalm']

logger = logging.getLogger(__name__)

def foo(grains):
    logger.debug("foo")
    logger.debug(grains)
    return {'interfaces': 'test_value'}

but after syncing:

# salt <proxy-minion-id> grains.item interfaces
<proxy-minion-id>:
    interfaces:
        - Loopback
        ...

and in logs from /var/log/salt/proxy

2019-09-19 15:53:27,624 [salt.loaded.ext.grains.issue                                  :8   ][DEBUG   ][23652] foo
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.issue                                  :9   ][DEBUG   ][23652] {u'kernel': u'proxy', u'saltpath': '/usr/lib/python2.7/site-packages/salt', u'osrelease': u'proxy', u'zmqversion': '4.3.2', u'kernelrelease': u'proxy', u'pythonpath': ['/usr/bin', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/site-packages'], u'ip6_gw': False, u'saltversioninfo': [2019, 2, 0, 0], u'fqdns': [], u'pythonversion': [2, 7, 16, 'final', 0], u'pythonexecutable': '/usr/bin/python2', u'os_family': u'proxy', u'osrelease_info': (u'proxy',), u'gpus': [], u'osfinger': u'proxy-proxy', u'kernelversion': u'proxy', u'ps': u'ps -efHww', u'locale_info': {}, u'ip4_gw': u'xxx.xxx.xxx.xxx', u'ip_gw': True, u'num_gpus': 0, u'saltversion': u'2019.2.0', u'hwaddr_interfaces': {u'lo': u'xx:xx:xx:xx:xx:xx', u'eth0': u'xx:xx:xx:xx:xx:xx'}, u'virtual': u'physical', u'osarch': 'x86_64', u'cpuarch': 'x86_64', u'osfullname': u'proxy', u'master': u'localhost', u'dns': {u'domain': u'', u'sortlist': [], u'nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'ip4_nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'search': [], u'ip6_nameservers': [], u'options': []}, u'path': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'os': u'proxy', u'mem_total': 0}
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.test_grains                            :10  ][DEBUG   ][23652] core_foo
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.test_grains                            :11  ][DEBUG   ][23652] {u'kernel': u'proxy', u'saltpath': '/usr/lib/python2.7/site-packages/salt', u'osrelease': u'proxy', u'zmqversion': '4.3.2', u'kernelrelease': u'proxy', u'pythonpath': ['/usr/bin', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/site-packages'], u'ip6_gw': False, u'saltversioninfo': [2019, 2, 0, 0], u'fqdns': [], u'pythonversion': [2, 7, 16, 'final', 0], u'pythonexecutable': '/usr/bin/python2', u'os_family': u'proxy', u'osrelease_info': (u'proxy',), u'gpus': [], u'osfinger': u'proxy-proxy', u'kernelversion': u'proxy', u'ps': u'ps -efHww', u'locale_info': {}, u'ip4_gw': u'xxx.xxx.xxx.xxx', u'ip_gw': True, u'num_gpus': 0, u'saltversion': u'2019.2.0', u'hwaddr_interfaces': {u'lo': u'xx:xx:xx:xx:xx:xx', u'eth0': u'xx:xx:xx:xx:xx:xx'}, u'virtual': u'physical', u'osarch': 'x86_64', u'cpuarch': 'x86_64', u'osfullname': u'proxy', u'master': u'localhost', u'dns': {u'domain': u'', u'sortlist': [], u'nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'ip4_nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'search': [], u'ip6_nameservers': [], u'options': []}, u'path': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'os': u'proxy', 'interfaces': 'test_value', u'mem_total': 0}
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.test_grains                            :10  ][DEBUG   ][23652] foo
2019-09-19 15:53:27,625 [salt.loaded.ext.grains.test_grains                            :11  ][DEBUG   ][23652] {u'kernel': u'proxy', u'saltpath': '/usr/lib/python2.7/site-packages/salt', u'osrelease': u'proxy', u'zmqversion': 2, u'kernelrelease': u'proxy', u'pythonpath': ['/usr/bin', '/usr/lib/python27.zip', '/usr/lib/python2.7', '/usr/lib/python2.7/plat-linux2', '/usr/lib/python2.7/lib-tk', '/usr/lib/python2.7/lib-old', '/usr/lib/python2.7/lib-dynload', '/usr/lib/python2.7/site-packages'], u'ip6_gw': False, u'saltversioninfo': [2019, 2, 0, 0], u'fqdns': [], u'pythonversion': [2, 7, 16, 'final', 0], u'pythonexecutable': '/usr/bin/python2', u'os_family': u'proxy', u'osrelease_info': (u'proxy',), u'gpus': [], u'osfinger': u'proxy-proxy', u'id': 'ios_xr_1', u'kernelversion': u'proxy', u'ps': u'ps -efHww', u'locale_info': {}, u'ip4_gw': u'xxx.xxx.xxx.xxx', u'ip_gw': True, u'num_gpus': 0, u'saltversion': u'2019.2.0', u'hwaddr_interfaces': {u'lo': u'xx:xx:xx:xx:xx:xx', u'eth0': u'xx:xx:xx:xx:xx:xx'}, u'virtual': u'physical', u'osarch': 'x86_64', u'cpuarch': 'x86_64', u'osfullname': u'proxy', u'master': u'localhost', u'dns': {u'domain': u'', u'sortlist': [], u'nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'ip4_nameservers': [u'xxx.xxx.xxx.xxx', u'xxx.xxx.xxx.xxx'], u'search': [], u'ip6_nameservers': [], u'options': []}, u'path': '/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin', u'os': u'proxy', 'interfaces': 'test_value', u'mem_total': 0}

'interfaces' initially didn't exist but then was added the custom grain value of 'test_value' (which must have been subsequently overwritten as seen by output of salt <proxy-minion-id> grains.item interfaces command above)

Furthermore later in /var/log/salt/proxy we see

2019-09-19 15:53:27,827 [netmiko                                                       :463 ][DEBUG   ][23652] read_channel:
2019-09-19 15:53:27,828 [netmiko                                                       :385 ][DEBUG   ][23652] write_channel: <?xml version="1.0" encoding="UTF-8"?><Request MajorVersion="1" MinorVersion="0"><Get><Operational><SystemTime/><PlatformInventory/>        </Operational></Get></Request>
2019-09-19 15:53:31,434 [netmiko                                                       :463 ][DEBUG   ][23652] read_channel: <?xml version="1.0" encoding="UTF-8"?>
<Response MajorVersion="1" MinorVersion="0"><Get><Operational><SystemTime MajorVersion="3" MinorVersion="0"><Clock><Year>2019</Year><Month>9</Month><Day>19</Day><Hour>15</Hour><Minute>45</Minute><Second>11</Second><Millisecond>247</Millisecond><Wday>4</Wday><TimeZone>UTC</TimeZone>
...
</Get><ResultSummary ErrorCount="0"/></Response>
XML>

Showing the xml being requested and responded to (likely from the core salt-napalm grains)

This isn't the behavior of the more 'standard' core grains:

# /srv/salt/_grains/test_grains.py
__proxyenabled__ = ['napalm']

def core_foo(grains):
     return {'zmqversion': 2}

after syncing:

# salt <proxy-minion-id> grains.item zmqversion
<proxy-minion-d>:
    zmqversion:
        2

Versions Report

bash-4.4$ salt --versions-report
Salt Version:
Salt: 2019.2.0

Dependency Versions:
cffi: 1.12.3
cherrypy: unknown
dateutil: 2.7.2
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
ioflo: Not Installed
Jinja2: 2.9.6
libgit2: 0.27.7
libnacl: Not Installed
M2Crypto: 0.33.0
Mako: Not Installed
msgpack-pure: Not Installed
msgpack-python: 0.5.6
mysql-python: Not Installed
pycparser: 2.17
pycrypto: 2.6.1
pycryptodome: Not Installed
pygit2: 0.27.4
Python: 2.7.16 (default, May 6 2019, 19:35:26)
python-gnupg: 0.4.2
PyYAML: 5.1.1
PyZMQ: 18.1.0
RAET: Not Installed
smmap: Not Installed
timelib: Not Installed
Tornado: 5.1.1
ZMQ: 4.3.2

System Versions:
dist:
locale: UTF-8
machine: x86_64
release: 3.10.0-862.3.2.el7.x86_64
system: Linux
version: Not Installed

Start with salt/grains/napalm.py and the custom _grains/issue.py example, then reproduce the behavior on a napalm proxy minion using grains.item interfaces. Compare the grain loading order shown in the logs with the documented precedence; done means a custom interfaces grain remains effective instead of being overwritten by the napalm grain.