[BUG] Grain loading precedence documentation is inconsistent

Description

We're using Salt 3002.2 and trying to use a custom grain to declare meta_role for a machine, based on a static role grain, but are unable to do so because of the order in which grains are processed. At the time of evaluation, /etc/salt/grains has not yet been loaded and so it is not passed in the grains argument to the function.

In reading the documentation for grains, this behavior is called out here:

For custom grains, if the function takes an argument grains, then the previously rendered grains will be passed in. Because the rest of the grains could be rendered in any order, the only grains that can be relied upon to be passed in are core grains.

However, just above that, the documentation says:

The order of evaluation is as follows:
Core grains.
Custom grains in /etc/salt/grains.
Custom grains in /etc/salt/minion.
Custom grain modules in _grains directory, synced to minions.

Similarly, grains from /etc/salt/minion override both core grains and custom grain modules, and grains in _grains will override any grains of the same name.

If I modify the example below to return role, I have confirmed that it does not override the value of role in /etc/salt/grains, because the file is loaded after grain modules in the _grains directory have been evaluated. I have tried putting a static file in that directory but it is not synced, seemingly confirming that only custom grain modules are expected in the _grains directory, and so the custom modules should have access to, and be able to override, grains defined in /etc/salt/grains, per the documentation.

Setup

Grain role on a machine, a list stored in /etc/salt/grains. In this example we could say the value could be one or more of [public_service, backend_service, postgres, redis].

# /etc/salt/grains
role:
- public_service

The meta_role grain code would look something like this:

# _grains/meta_role.py

meta_role_map = dict(
  application=['public_service', 'backend_service'],
  database=['postgres', 'redis']
)

def main(grains=None):
  grains = grains or {}
  meta_roles = set()
  roles = grains.get('role', [])

  for role in roles:
    for meta_role, mapped_roles in meta_role_map.items():
      if role in mapped_roles:
        meta_roles.add(meta_role)

  return dict(
    meta_role=list(meta_roles)
  )

Steps to Reproduce the behavior

After declaring the above static and dynamic grains, and syncing them to a minion, run the following commands:

sudo salt-call grains.get role
sudo salt-call grains.get meta_role

Expected behavior

I expect that grains.get role returns the list [public_service], and grains.get meta_role returns the list [application]. However, meta_role returns an empty list because /etc/salt/grains has not been loaded at the time of the _grains/meta_role.py evaluation.

Versions Report

Salt Version:
          Salt: 3002.2

Dependency Versions:
          cffi: Not Installed
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: 4.0.7
     gitpython: 3.1.8
        Jinja2: 2.10.3
       libgit2: Not Installed
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.0
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     pycparser: Not Installed
      pycrypto: Not Installed
  pycryptodome: 3.4.7
        pygit2: Not Installed
        Python: 3.5.2 (default, Nov 23 2017, 16:37:01)
  python-gnupg: 0.3.8
        PyYAML: 5.3.1
         PyZMQ: 17.1.2
         smmap: 4.0.0
       timelib: Not Installed
       Tornado: 4.5.3
           ZMQ: 4.1.4

System Versions:
          dist: ubuntu 16.04 Xenial Xerus
        locale: UTF-8
       machine: x86_64
       release: 4.4.0-87-generic
        system: Linux
       version: Ubuntu 16.04 Xenial Xerus

Additional context

It appears that this bug was described previously in 2018, and recently closed due to inactivity https://github.com/saltstack/salt/issues/50491

Start with the grains precedence documentation and the custom module shown at _grains/meta_role.py, then reproduce the behavior with /etc/salt/grains and the two salt-call commands. Compare the documented evaluation order with the observed meta_role result and determine whether the documentation or grain-loading behavior is inconsistent. Done means the precedence is accurately documented or the reported behavior is corrected and covered by an appropriate test.