[BUG] state.apply fails if pillar uses custom grain from _grains

Description

state.apply fails and does not sync custom grains and modules scripts from /srv/salt/_grains and and /srv/salt/_modules directories if a pillar uses the custom grain or custom execution module.

Setup

Please be as specific as possible and give set-up details.

• VM (KVM) running on AWS ec2 instance.
• onedir packaging
• masterless

Steps to Reproduce the behavior

• Setup directories and files; set minion to masterless; create a custom grain script and a pillar that uses custom grain returned from the script:

  cat > /etc/salt/minion <<'EOF'
  file_client: local
  master_type: disable
  EOF
  
  mkdir /srv/salt /srv/pillar /srv/salt/_grains
  cat > /srv/salt/top.sls <<'EOF'
  base:
    '*':
      - test
  EOF
  
  cat > /srv/salt/test.sls <<'EOF'
  "do nothing":
    test.nop: []
  EOF
  
  cat > /srv/salt/_grains/custom_grain.py <<'EOF'
  def main():
      return {'custom_grain': 'test_value'}
  EOF
  
  cat > /srv/pillar/top.sls <<'EOF'
  base:
    '*':
      - defaults
  EOF
  
  cat > /srv/pillar/defaults.sls <<'EOF'
  mypillar: "{{ grains['custom_grain'] }}"
  EOF
  
  

• Run state.apply which results in an error

salt-call --local state.apply

salt.exceptions.SaltRenderError: Jinja variable 'dict object' has no attribute 'custom_grain'; line 1

---
mypillar: "{{ grains['custom_grain'] }}"    <======================

---
[CRITICAL] Pillar render error: Rendering SLS 'defaults' failed. Please see master log for details.
local:
    Data failed to compile:
--------
    Pillar failed to render with the following messages:
--------
    Rendering SLS 'defaults' failed. Please see master log for details.

Expected behavior
The custom _grains and _modules should be synced before rendering the pillars and state.apply should run successfully to completion.

Screenshots
If applicable, add screenshots to help explain your problem.

Versions Report

Salt Version:
    Salt: 3006.2

Python Version:
    Python: 3.10.12 (main, Aug 3 2023, 21:47:10) [GCC 11.2.0]

Dependency Versions:
    cffi: 1.14.6
    cherrypy: 18.6.1
    dateutil: 2.8.2
    Jinja2: 3.1.2
    msgpack: 1.0.2
    packaging: 22.0
    pycparser: 2.21
    pycryptodome: 3.9.8
    python-gnupg: 0.4.8
    PyYAML: 6.0.1
    PyZMQ: 23.2.0
    relenv: 0.13.3
    timelib: 0.2.4
    Tornado: 4.5.3
    ZMQ: 4.3.4

System Versions:
    dist: rhel 8.8 Ootpa
    locale: utf-8
    machine: x86_64
    release: 4.18.0-477.13.1.el8_8.x86_64
    system: Linux
    version: Red Hat Enterprise Linux 8.8 Ootpa

Additional context
Running salt-call --local saltutil.sync_all --pillar-root=/dev/null before running state.apply syncs the _grains and _modules correctly and allows the state.apply to run correctly.

Reproduce the failure with the provided /etc/salt/minion, /srv/salt/_grains, /srv/salt/_modules, and /srv/pillar setup, then compare state.apply with the saltutil.sync_all workaround. Trace the ordering between syncing custom extensions and pillar rendering; done when state.apply syncs the extensions before rendering and completes successfully.