Razor server crash when trying to analyse deleted files

## Issue Description ##
After removing or renaming a controller and views, Razor language server crash at VS code startup

## Steps to Reproduce ##
1. open project (ASP.Net MVC core 3.1),
2. rename a controller (class, filename) and rename the view folder.
3. restart VS code
4 Razor language server crash 3 time and won't restart

## Expected Behavior ##
No crash and RLS should not try to analyse deleted files
## Actual Behavior ##
Razor try to read deleted files and crash
## Logs ##

I got this error in the Razor output:
Unhandled exception. System.IO.DirectoryNotFoundException: Could not find a part of the path '/Users/xxxx/xxxx/xxx/Views/xxx/xxxx.cshtml'.
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at Microsoft.AspNetCore.Razor.LanguageServer.Common.DefaultRemoteTextLoaderFactory.RemoteTextLoader.LoadTextAndVersionAsync(Workspace workspace, DocumentId documentId, CancellationToken cancellationToken) in /_/src/Razor/src/Microsoft.AspNetCore.Razor.LanguageServer.Common/DefaultRemoteTextLoaderFactory.cs:line 62
at Microsoft.CodeAnalysis.Razor.ProjectSystem.DefaultProjectSnapshotManager.<>c__DisplayClass17_0.b__0() in /_/src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/ProjectSystem/DefaultProjectSnapshotManager.cs:line 152
at Microsoft.CodeAnalysis.Razor.ProjectSystem.DocumentState.GetTextVersionAsync() in /_/src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/ProjectSystem/DocumentState.cs:line 127
at Microsoft.CodeAnalysis.Razor.ProjectSystem.DocumentState.ComputedStateTracker.GetGeneratedOutputAndVersionCoreAsync(DefaultProjectSnapshot project, DocumentSnapshot document) in /_/src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/ProjectSystem/DocumentState.cs:line 352
at Microsoft.CodeAnalysis.Razor.ProjectSystem.DefaultDocumentSnapshot.GetGeneratedOutputAsync() in /_/src/Razor/src/Microsoft.CodeAnalysis.Razor.Workspaces/ProjectSystem/DefaultDocumentSnapshot.cs:line 61
at Microsoft.AspNetCore.Razor.LanguageServer.RazorDiagnosticsPublisher.PublishDiagnosticsAsync(DocumentSnapshot document) in /_/src/Razor/src/Microsoft.AspNetCore.Razor.LanguageServer/RazorDiagnosticsPublisher.cs:line 161
at Microsoft.AspNetCore.Razor.LanguageServer.RazorDiagnosticsPublisher.WorkTimer_Tick(Object state) in /_/src/Razor/src/Microsoft.AspNetCore.Razor.LanguageServer/RazorDiagnosticsPublisher.cs:line 204
at System.Threading.Tasks.Task.<>c.b__139_1(Object state)
at System.Threading.QueueUserWorkItemCallbackDefaultContext.Execute()
at System.Threading.ThreadPoolWorkQueue.Dispatch()
at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()
[Info - 10:08:11 AM] Connection to server got closed. Server will restart.

### OmniSharp log ###
nothing special in omnisharp log, no error just projects loading trace

### C# log ###
Post the output from Output-->C# here

## Environment information ##

**VSCode version**: 1.46.1
**C# Extension**: 1.22.1

Mono Information
OmniSharp using global mono :6.8.0
Dotnet Information
.NET Core SDK (reflecting any global.json):
Version: 3.1.200
Commit: c5123d973b

Runtime Environment:
OS Name: Mac OS X
OS Version: 10.15
OS Platform: Darwin
RID: osx.10.15-x64
Base Path: /usr/local/share/dotnet/sdk/3.1.200/

Host (useful for support):
Version: 3.1.2
Commit: 916b5cba26

.NET Core SDKs installed:
3.0.100 [/usr/local/share/dotnet/sdk]
3.1.100 [/usr/local/share/dotnet/sdk]
3.1.101 [/usr/local/share/dotnet/sdk]
3.1.200 [/usr/local/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.App 3.0.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.0 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.1 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.13 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.14 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.15 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.16 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.0.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.0 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.1 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download

Visual Studio Code Extensions

|Extension|Author|Version|
|---|---|---|
|aspnet-helper|schneiderpat|0.6.4|
|auto-close-tag|formulahendry|0.5.7|
|auto-rename-tag|formulahendry|0.1.3|
|azure-account|ms-vscode|0.8.11|
|csharp|ms-dotnettools|1.22.1|
|Csharp-ASPNETCore|rahulsahay|1.11.0|
|docomment|k--kato|0.1.11|
|dotnet-test-explorer|formulahendry|0.7.3|
|flow-for-vscode|flowtype|1.5.0|
|html-preview-vscode|tht13|0.2.5|
|live-sass|ritwickdey|3.0.0|
|LiveServer|ritwickdey|5.6.1|
|mssql|ms-mssql|1.9.0|
|open-html-in-browser|peakchen90|2.1.3|
|path-intellisense|christian-kohler|2.2.1|
|todo-tree|Gruntfuggly|0.0.177|
|vscode-colorize|kamikillerto|0.8.17|
|vscode-css-formatter|aeschli|1.0.1|
|vscode-docker|ms-azuretools|1.3.1|
|vscode-eslint|dbaeumer|2.1.5|
|vscode-html-css|ecmel|0.2.3|
|vscode-icon-theme|jtlowe|1.6.6|
|vscode-intelephense-client|bmewburn|1.4.1|
|vscode-react-native|msjsdiag|0.16.0|
|vscodeintellicode|VisualStudioExptTeam|1.2.8|;