compiler-explorer / compiler-explorer/infra
Optimize library installer conan calls
- Dominant language
- Python
- Stars
- 434
- Forks
- 429
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 38
Description
Now that the Cloudfront caching in front of the Conan server is disabled, we need to figure out a way to manually optimize the process.
A large IO bottleneck is the checks that we do to prevent from building the same thing over and over again:
* `has_failed_before` https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L783
* `is_already_uploaded` https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L796
- `get_build_annotations` https://github.com/compiler-explorer/infra/blob/main/bin/lib/library_builder.py#L752 is part of this
These are called for every compiler that will be built with. There might be a way to request the information for all compilers at once and then only do the lookup per compiler.
Related endpoint: https://github.com/compiler-explorer/conanproxy/blob/main/index.js#L454
* See also: https://github.com/compiler-explorer/conanproxy/blob/main/build-annotations.js#L18
* Unfortunately the conan_hash will be unique per compiler, but if you iterate over all the directories in the package directory (https://github.com/compiler-explorer/conanproxy/blob/main/build-annotations.js#L11C102-L11C111) you can match the hashes on the builder side
Lesser influence, but still IO:
* `get_commit_hash` https://github.com/compiler-explorer/infra/blob/85647382536eeb1284845497b94850cb8c8427be/bin/lib/library_builder.py#L769
- This really only has to be called once, so either the first time its called or just in the constructor
Contributor guide
Research direction
Start in bin/lib/library_builder.py by reading has_failed_before, is_already_uploaded, get_build_annotations, and get_commit_hash. Compare these calls with the conanproxy endpoint in index.js and build-annotations.js; done means reducing repeated IO while preserving the per-compiler hash matching and calling get_commit_hash only once.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- infrastructure, performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100