Minion files cache, does not delete files removed from the file roots defined on the master

Description of Issue

Files within the 'file_roots' that have been cached on a minion are not deleted once they become stale (deleted on the master) and can still be accessed at least by jinja templating.

ie, files in minion cache that have been deleted from the master are still used,
at least by jinja templating - the cached deleted files are not removed, and
jinja templating which incorrectly 'includes' the deleted files still succeeds in accessing the 'stale' cached files, instead of failing.

On my master, I had a structure like this:

/srv/salt/state/templates/example/example.sls

The files was then renamed to just:

/srv/salt/state/templates/example.sls

The directory /srv/salt/state/templates/example
and all of its contents were then removed.

I have another state file which uses jinja to do an include.
It is including the file like this:

{% include('templates/example/example.sls') with context %}

I found that although the entire templates/example/example
directory was removed, when the states were executed, the
jinja include successfully loads and processes the cached
file within /var/cache/salt/minion/files/base/

Setup

example setup would be:

set within the master config file (ie within /etc/salt/master):

fileserver_backend:
  - roots
file_roots:
  base:
     - /srv/salt/state

create a template file, /srv/salt/state/templates/example/example.sls, with some content such as:

example:
    cmd.run:
       - name: echo before deletion

create a file, /srv/salt/state/example.sls, with some content such as:

{% include('templates/example/example.sls') %}

Steps to Reproduce Issue

then run the state on a minion:

salt minion-name state.sls example

It should display that the command 'echo before deletion' has been run.

Next, move /srv/salt/templates/example/example.sls to /srv/salt/templates/example.sls, and then edit it to change the command to echo something different:

example:
    cmd.run:
       - name: echo after deletion

At this point, do NOT update the file /srv/salt/example.sls to refer to the new template location.

Run the state again:

salt minion-name state.sls example

The output should show either show an error, due to the failure to include the deleted file, or would perhaps silently fail to include the file.

Instead, it successfully executes a state, based on processing the old file which is still in the minion cache, and it still displays that the command 'echo before deletion' was run.

Although it is a developer error, that they failed to update the jinja include, this error is hidden because the stale file is kept around in the minion file cache.

Versions Report

Salt Version:
           Salt: 2019.2.0
 
Dependency Versions:
           cffi: 1.6.0
       cherrypy: unknown
       dateutil: 1.5
      docker-py: Not Installed
          gitdb: Not Installed
      gitpython: Not Installed
          ioflo: Not Installed
         Jinja2: 2.7.2
        libgit2: Not Installed
        libnacl: Not Installed
       M2Crypto: Not Installed
           Mako: Not Installed
   msgpack-pure: Not Installed
 msgpack-python: 0.5.6
   mysql-python: Not Installed
      pycparser: 2.14
       pycrypto: 2.6.1
   pycryptodome: Not Installed
         pygit2: Not Installed
         Python: 2.7.5 (default, Oct 30 2018, 23:45:53)
   python-gnupg: Not Installed
         PyYAML: 3.11
          PyZMQ: 15.3.0
           RAET: Not Installed
          smmap: Not Installed
        timelib: Not Installed
        Tornado: 4.2.1
            ZMQ: 4.1.4
 
System Versions:
           dist: centos 7.6.1810 Core
         locale: UTF-8
        machine: x86_64
        release: 3.10.0-957.el7.x86_64
         system: Linux
        version: CentOS Linux 7.6.1810 Core

workaround for users encountering this issue

deleting the minion caches with the following command may help:

salt '*' saltutil.clear_cache

however I cannot verify for sure this as I already deleted my caches via rm -rf /var/cache/salt/minion/files/base/*

Reproduce the scenario using the master file_roots setup and the minion command shown, then inspect how files are cached under /var/cache/salt/minion/files/base/. Compare the stale include behavior with the expected failure after the master file is moved, using saltutil.clear_cache only as the documented workaround; done means deleted files are no longer served from the minion cache.