Keeps downloading Omnisharp

## Environment data
`dotnet --info` output:

.NET Core SDK (reflecting any global.json):
Version: 3.1.101
Commit: b377529961

Runtime Environment:
OS Name: ubuntu
OS Version: 18.04
OS Platform: Linux
RID: ubuntu.18.04-x64
Base Path: /usr/share/dotnet/sdk/3.1.101/

Host (useful for support):
Version: 3.1.1
Commit: a1388f194c

.NET Core SDKs installed:
2.1.300-rc1-008673 [/usr/share/dotnet/sdk]
2.1.804 [/usr/share/dotnet/sdk]
3.1.101 [/usr/share/dotnet/sdk]

.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.0-rc1-final [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.16 [/usr/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.0-rc1-final [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.16 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.1 [/usr/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.0-rc1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.16 [/usr/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.1 [/usr/share/dotnet/shared/Microsoft.NETCore.App]

VS Code version: 1.46.1
C# Extension version: v1.22.1

## OmniSharp log
## Steps to reproduce

- (re)Start VSC
- Open a .cs file

## Expected behavior

- allowing me to edit/run/debug with all features enabled.

## Actual behavior

- It starts download omnisharp/debugger etc.

This issue has been reported here before but the 'fixes' there did not help. I'm not behind a proxy (I am reporting this on another connection than my home connection for instance and it happens here too) and it happens on win/lin/mac (all the same VSC version and dotnet runtime; I develop on all 3).

My connection is fast enough so it's not a real bother, but it seems a bug as it definitely did download before and it works fine once downloaded, so why does it download again?