Processing a large compile_commands.json takes too long

Environment

• OS and Version: Win10 22H2 19045
• VS Code Version: 1.94
• C/C++ Extension Version: 1.21.6
• If using SSH remote, specify OS of remote machine: N/A

Bug Summary and Steps to Reproduce

Bug Summary: The extension gets stuck initializing Intellisense more or less permanently when loading a compilation database with a large number of file entries. The extension is unusable while this occurs. It's a single threaded process that appears to process every file in the compilation database before enabling any LSP features for currently opened files.

Steps to reproduce:

1. Clone UnrealEngine (use a shallow clone to make it faster)
   a. Sorry, I know this is a pain for a repro. There's a small, self-contained partial repro I can't share publicly if there's a private channel available.
2. Open the folder with vscode
3. Run GenerateProjectFiles.bat -VSCode
4. Close and re-open vscode (still as a folder, you don't need to use the workspace)
5. Open a file (e.g. AssertionMacroTests.cpp)
6. Notice Intellisense is initializing
7. Wait a while
8. Notice Intellisense is still initializing
9. Attempt to "go to definition" on a #include statement
10. Notice no code navigation or code completion operations work

Expected behavior:

• Intellisense and LSP operations for the currently active file are prioritized above all other files.
• Beginning normal LSP operations is not blocked on doing any operation on every single file in the directory.

When not using compile commands the tag parser fallback behaves correctly. It will immediately parse the current open file. Similarly, clangd handles the current file before doing background indexing.

It's also possible to observe the bad behavior with just the .vscode directory and not the entire engine. When the actual code files are missing the C/C++ Configuration Warnings output will be spammed with errors like this:

[10/9/2024, 9:37:04 AM] Unable to locate forced include: D:\UnrealEngine\Engine\Intermediate\Build\Win64\x64\UnrealEditorGPF\Development\UnrealEd\SharedPCH.UnrealEd.Cpp20.h

While this is occurring, Intellisense is in the "initializing" state indicating that it is processing every single file in the database before the init completes and LSP operations in the current file begin to function.

Configuration and Logs

{
    "configurations": [
        {
            "name": "Win32",
            "compilerPath": "C:\\Program Files\\Microsoft Visual Studio\\2022\\Professional\\VC\\Tools\\MSVC\\14.38.33130\\bin\\Hostx64\\x64\\cl.exe",
            "cStandard": "c17",
            "cppStandard": "c++20",
            "intelliSenseMode": "msvc-x64",
            "compileCommands": "D:\\UnrealEngine\\.vscode\\compileCommands_Default.json"
        }
    ],
    "version": 4
}

I don't think I can get C/C++: Log Diagnostics. The command doesn't do anything. Looks like it's an LSP request? And since the LSP is essentially stalled it's not returning a response.

loggingLevel: Debug
LSP: (received) cpptools/initialize (id: 1)
LSP: (invoked) cpptools/initialize (id: 1)
cpptools version (TypeScript): 1.21.6
cpptools version (native): 1.21.6.0
Autocomplete is enabled.
Error squiggles are enabled.
Hover is enabled.
IntelliSense Engine = default.
LSP: Sending response (id: 1)
LSP: (received) cpptools/queryCompilerDefaults (id: 2)
LSP: (invoked) cpptools/queryCompilerDefaults (id: 2)
LSP: Sending response (id: 2)
LSP: (received) cpptools/didChangeCppProperties (id: 3)
LSP: (invoked) cpptools/didChangeCppProperties (id: 3)
LSP: (received) cpptools/didChangeSettings
LSP: (received) cpptools/didChangeSettings
LSP: (received) cpptools/didChangeSettings
LSP: (received - deferred) textDocument/didOpen: file:///d%3A/UnrealEngine/Engine/Source/Runtime/Core/Tests/Misc/AssertionMacrosTest.cpp
LSP: Message ignored due to no registered handler: $/setTrace
LSP: (received - deferred) cpptools/didChangeVisibleTextEditors
LSP: (received - deferred) cpptools/didChangeVisibleTextEditors
LSP: (received - deferred) cpptools/didChangeVisibleTextEditors
LSP: (received - deferred) cpptools/getCodeActions: file:///d%3A/UnrealEngine/Engine/Source/Runtime/Core/Tests/Misc/AssertionMacrosTest.cpp (id: 4)
LSP: (received - deferred) cpptools/getCodeActions: file:///d%3A/UnrealEngine/Engine/Source/Runtime/Core/Tests/Misc/AssertionMacrosTest.cpp (id: 5)
LSP: (received - deferred) cpptools/getFoldingRanges: file:///d%3A/UnrealEngine/Engine/Source/Runtime/Core/Tests/Misc/AssertionMacrosTest.cpp (id: 6)
LSP: (received) cpptools/didChangeSettings
LSP: (received - deferred) cpptools/getCodeActions: file:///d%3A/UnrealEngine/Engine/Source/Runtime/Core/Tests/Misc/AssertionMacrosTest.cpp (id: 7)
LSP: (received - deferred) cpptools/getCodeActions: file:///d%3A/UnrealEngine/Engine/Source/Runtime/Core/Tests/Misc/AssertionMacrosTest.cpp (id: 8)
LSP: $/cancelRequest (cpptools/getCodeActions, id: 4)
LSP: $/cancelRequest (cpptools/getCodeActions, id: 5)
LSP: $/cancelRequest (cpptools/getFoldingRanges, id: 6)
LSP: $/cancelRequest (cpptools/getCodeActions, id: 7)
LSP: (received - deferred) cpptools/getDocumentSymbols: file:///d%3A/UnrealEngine/Engine/Source/Runtime/Core/Tests/Misc/AssertionMacrosTest.cpp (id: 9)
Code browsing service initialized
Attempting to get defaults from C++ compiler in "compilerPath" property: 'C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\cl.exe'
Attempting to get defaults from C compiler in "compilerPath" property: 'C:\Program Files\Microsoft Visual Studio\2022\Professional\VC\Tools\MSVC\14.38.33130\bin\Hostx64\x64\cl.exe'

Other Extensions

I've repro'd on Insiders with no other extensions.

Additional context

No response

Reproduce with the Unreal Engine compile_commands.json configuration and inspect the LSP initialization sequence, especially cpptools/initialize, textDocument/didOpen, and the deferred requests shown in the logs. Trace how the compilation database is processed relative to the currently active file. Done means the active file supports IntelliSense and LSP operations without waiting for every database entry to finish.