compiler-explorer / compiler-explorer/infra

Optimize library installer conan calls

Open
#1,342 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
434
Forks
429
Avg merge
4d 21h
Merged PRs (30d)
38

Description

Now that the Cloudfront caching in front of the Conan server is disabled, we need to figure out a way to manually optimize the process.

A large IO bottleneck is the checks that we do to prevent from building the same thing over and over again:
* `has_failed_before` https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L783
* `is_already_uploaded` https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L796
- `get_build_annotations` https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L752 is part of this

These are called for every compiler that will be built with. There might be a way to request the information for all compilers at once and then only do the lookup per compiler.

Related endpoint: https://github.com/compiler-explorer/conanproxy/blob/main/index.js#L454
* See also: https://github.com/compiler-explorer/conanproxy/blob/main/build-annotations.js#L18
* Unfortunately the conan_hash will be unique per compiler, but if you iterate over all the directories in the package directory (https://github.com/compiler-explorer/conanproxy/blob/main/build-annotations.js#L11C102-L11C111) you can match the hashes on the builder side

Lesser influence, but still IO:
* `get_commit_hash` https://github.com/compiler-explorer/infra/blob/85647382536eeb1284845497b94850cb8c8427be/bin/lib/library_builder.py#L769
- This really only has to be called once, so either the first time its called or just in the constructor

Contributor guide

Open the contributing guide

Research direction

Start in bin/lib/library_builder.py by reading has_failed_before, is_already_uploaded, get_build_annotations, and get_commit_hash. Compare these calls with the conanproxy endpoint in index.js and build-annotations.js; done means reducing repeated IO while preserving the per-compiler hash matching and calling get_commit_hash only once.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
infrastructure, performance
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.