github / github/codeql

Using binary logs to significantly increase CodeQL analysis performance for C#

未關閉
#16,346 25 則留言 7 個 reaction 已指派 1 人 已被 @tamasvajk 認領 在 GitHub 檢視
C# question
主要語言
CodeQL
星號
10.1k
分支
2.1k
平均合併
2 天 15 小時
30 天內合併 PR
141

描述

The nature of the C# driver means that it needs to see all csc invocations for a build. Today that is achieved by [disabling shared compilation][shared] during build. That unfortunately creates a significant performance penalty when building C# code. The larger the repo the more significant the penalty. The rule of thumb is that for solutions of at least medium size this will cause somewhere from a 3-4X build slow down. For larger repositories this can be even bigger.

Consider as a concrete example the [dotnet/runtime][runtime] repository. When their CodeQL pipeline runs and shared compilation is disabled it increases their build time by 1 hour and 38 minutes. Or alternatively building the [dotnet/roslyn][roslyn] repository. Building that locally results in a ~4X perf penalty when shared compilation is disabled.

I understand from [previous conversations][convo] that this is done because the extractor needs to see every C# compiler call in the build and potentially process it. An alternative approach to doing this is to have the build produce a binary log. That is a non-invasive change to builds and there is ample tooling for reading C# compiler calls from those once the build completes. The code for reading compiler calls is at the core very straight forward:

```csharp
using var fileStream = new FileStream(binlogPath, FileMode.Open, FileAccess.Read, FileShare.Read);
List compilerCalls = BinaryLogUtil.ReadAllCompilerCalls(fileStream);
```

I have a [sample][sample] here of integrating this approach into the existing extractor tool. On my machine I was able to use this sample to run over a full build of [roslyn][roslyn] without any obvious issues. And most importantly, no changes to how I built roslyn :smile:

Using libraries like this have other advantages because you could also offload the work of creating `CSharpCompilation` objects. There is code in that library that will very faithfully recreate a `CSharpCompilation` for every `CompilerCall` instance. Furthermore, think there are other optimization opportunities for the extractor once you are running the analysis in bulk: caching `MetadataReferences`, caching `SourceText`, etc ...

More than happy to chat about this, the different ways binary logs can be used, moving analysis off machine, etc ... My big end goal is to get back to us using shared compilation so we can keep our build times down.

[shared]: https://github.com/github/codeql/pull/9705
[runtime]: https://github.com/dotnet/runtime
[roslyn]: https://github.com/dotnet/roslyn
[convo]: https://github.com/github/codeql/pull/9705#issuecomment-1580975144
[sample]: https://github.com/jaredpar/codeql/commit/9a306c1d1caba619275754a6eceba2b327383a0e

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。