SetPaketCommand takes half a second

Description

An empty dotnet core 3.1 web app project takes ~1 sec to build 2nd time (no changes). Half of this 1 sec is taken by SetPaketCommand, even when --no-restore flag is set. Should this target be executed at all in this case?

Repro steps

1. With dotnet core SDk 3.1, cd test-proj && dotnet new web.
2. Configure paket as a local dotnet tool, add dependencies and references files.
3. paket install, restore etc.
4. dotnet build .
5. dotnet build . --no-restore /clp:PerformanceSummary

The output has this:

Target Performance Summary:
        ...
        0 ms  Compile                                    1 calls
        0 ms  AfterResolveReferences                     1 calls
        0 ms  CoreBuild                                  1 calls
        0 ms  BeforeResGen                               1 calls
        ...
        0 ms  Build                                      1 calls
        ...
        3 ms  CoreCompile                                1 calls
        ...
       19 ms  PaketRestore                               1 calls
       36 ms  ResolveProjectReferences                   1 calls
       37 ms  GenerateBuildDependencyFile                1 calls
       38 ms  ProcessFrameworkReferences                 1 calls
       52 ms  FindReferenceAssembliesForReferences       1 calls
       53 ms  GenerateBuildRuntimeConfigurationFiles     1 calls
       55 ms  ResolveAssemblyReferences                  1 calls
       67 ms  ResolvePackageAssets                       1 calls
      461 ms  SetPaketCommand                            1 calls
...
Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:00:01.18

Expected behavior

SetPaketCommand should not take half the build time, preferably it should not be executed at all if --no-restore flag is used

Actual behavior

SetPaketCommand takes ~0.5 sec, which is ~50% of build time.

Known workarounds

N/A

Start by tracing the SetPaketCommand target during the dotnet build path, using the listed dotnet core 3.1 reproduction and /clp:PerformanceSummary output. Compare a normal build with dotnet build . --no-restore and determine whether SetPaketCommand still runs; done means the target no longer adds the reported delay when restore is disabled.