dmyy crashes when switching to git branches which removes subdirectory

Crash Report

I am using dmypy to check code in a pre-commit hook. When switching branches, obviously a lot of files/packages come and go. It seems as if dmypy has problems with the internal cache when a formerly checked file/package is not available anymore.
I have encountered two different backtraces.

Traceback

Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 221, in serve
  File "mypy/dmypy_server.py", line 264, in run_command
  File "mypy/dmypy_server.py", line 323, in cmd_run
  File "mypy/dmypy_server.py", line 385, in check
  File "mypy/dmypy_server.py", line 559, in fine_grained_increment_follow_imports
  File "mypy/dmypy_server.py", line 688, in direct_imports
KeyError: 'removed.package'

Daemon crashed!
Traceback (most recent call last):
  File "mypy/dmypy_server.py", line 221, in serve
  File "mypy/dmypy_server.py", line 264, in run_command
  File "mypy/dmypy_server.py", line 323, in cmd_run
  File "mypy/dmypy_server.py", line 385, in check
  File "mypy/dmypy_server.py", line 577, in fine_grained_increment_follow_imports
  File "mypy/server/update.py", line 1164, in refresh_suppressed_submodules
  File "mypy/fscache.py", line 172, in listdir
  File "mypy/fscache.py", line 168, in listdir
FileNotFoundError: [Errno 2] No such file or directory: '..../deleted/directory'

To Reproduce

Switch between git branches which add/remove a complete subdirectory.

Your Environment

• Mypy version used: 0.812
• dmypy command-line: dmypy run -- --config-file mypy.ini -p package1 -p package2 -p package3
• Python version used: 3.9
• Operating system and version: MacOS

Reproduce the failure by running the reported dmypy command, switching between branches that add or remove a complete subdirectory, and checking the daemon tracebacks. Start in mypy/dmypy_server.py at fine_grained_increment_follow_imports, direct_imports, and the calls into mypy/server/update.py and mypy/fscache.py. Done means branch changes no longer crash dmypy and the relevant regression behavior is covered.