dotnet / dotnet/arcade

SignTool rewrite for efficiency

Open
#11,066 0 comments 0 reactions 0 assignees View on GitHub
area-eng-services Epic
Dominant language
C#
Stars
729
Forks
397
Avg merge
3d 15m
Merged PRs (30d)
149

Description

## Description

SignTool is what we use to sign files within .NET builds based on Arcade. Given a set of input files, it determines what files need signing, what their certificates should be, and submits them to MicroBuild (which submits to ESRP) for signing. When files that should be signed are containers (zips, nupkgs, msis), SignTool has the ability to unpack those containers, sign the files within, and repack those. This works even recursively.

SignTool was originally designed for two purposes:
- Make signing simpler in a repo (don't need to inject signing processes in the middle of a build, just sign all the containers at the end_
- Make signing faster - There are often duplicate files within a package, or within a set of packages produced by a repo, or set of repos, etc. Deduplicating these files allows for fewer signing submissions, which improves efficiency.

SignTool was originally used on the scale of smallish repos. As repos have gotten larger, the algorithm it uses isn't quite as efficient as it could be. Today, sign tool works as follows:

1. Unpack all files, recursively, within the set of files passed to signing (e.g. nupkgs, msis, etc.. Dedupe as you go.
2. Given the entire set of files, determine the certificates for every fike.
3. Working from the innermost "layer", sign those files, then repack them in their parent containers, sign those, repack, etc. until the operation is complete.

While deduping is the major optimization that SignTool brings, the "stage" based approach isn't quite as efficient as it could be. Data from post-build signing passes in 6.0 reveals that signing itself takes ~2hrs of the total time, while unpacking, repacking, analysis, etc. brings that time up to ~3hrs 45 mins. There are signficant wins to be had.

The wins rely on eliding the unpack/repack/analysis time and the time spent waiting for ESRP. The time spent waiting on ESRP is I/O minimal (except for file upload/download), while the time spent doing unpack/repack is I/O heavy. In addition, the current client is not async/await enabled.

The basic idea, I think, would be to switch the "sign" stage to submit a set of files to Microbuild as soon as the file set was large enough, and to repack as soon as all files in a given container have their signing steps completed. Deduping would **still** take place, it would simply be deduping wouldn't happen all at once. If a file Foo.dll is signed, and then later as we unpack we discover another identical Foo.dll (the original file name and hash are the same), then we simply substitute in the alread-signed copy. Combining this, with a parallelize approach to unpack/pack (basically, pack/unpack would be parallel with N lanes to avoid overtaxing the I/O systems), I think you could get the Signing time relatively close to the "floor" of 2 hrs.

This would have **significant wins** even outside of post-build signing, in repos like runtime in 8.0 and 7.0 with in-build signing because they sign huge quantities of artifacts.

Note that this would mean a move away from Microbuild and onto ESRP's client/libraries, because Microbuild doesn't support this kind of thing (IIRC Microbuild is singleton and not multi-proc safe). This wuold also be good as it would mean that you could sign on non-Windows machines more easily.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.