trixi-framework / trixi-framework/Trixi.jl
Speed up testing using a custom sysimage for the dependencies
- Dominant language
- Julia
- Stars
- 731
- Forks
- 166
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 25
Description
I have an idea for speeding up the tests by cutting away at the part that is currently not parallelized: precompilation.
### TL;DR
Use PackageCompiler.jl to create sysimages for CI testing such that precompilation of dependencies becomes a non-issue.
### Long version
Right now, one of the time consuming parts for CI testing is that each CI container starts fresh, thus we incur the time for downloading, precompiling, and then actually compiling all dependencies for each separate test job. Given that we have quite a few dependencies in `Project.toml`,
https://github.com/trixi-framework/Trixi.jl/blob/13a3e580a148566f2076ca5d60fcfda6444d978a/Project.toml#L7-L27
and in `test/Project.toml`,
https://github.com/trixi-framework/Trixi.jl/blob/13a3e580a148566f2076ca5d60fcfda6444d978a/test/Project.toml#L2-L10
this is one of the reasons we can't further parallelize testing, since that precompilation part is inherently non-parallel.
Therefore, I had the following idea: We could use [PackageCompiler.jl](https://github.com/JuliaLang/PackageCompiler.jl) to create a custom sysimage for Julia using a separate CI script, and store them as [workflow artifacts](https://docs.github.com/en/actions/guides/storing-workflow-data-as-artifacts). We would bake in only the dependencies of Trixi, not Trixi itself, since packages compiled into a sysimage are [locked to their version used in the sysimage](https://julialang.github.io/PackageCompiler.jl/dev/sysimages/#Drawbacks-to-custom-sysimages-1). For the same reason, we would have to regenerate the sysimage each time either `Project.toml` or `test/Project.toml` changes, to make sure we have the correct versions.
### Challenges
1. We need some GitHub REST API Fu to download the correct sysimages as artifacts.
2. We need to robustly figure out when the `Project.toml` files change and update the sysimages accordingly.
3. Different PRs might live at different versions of the `Project.toml` files. Thus the sysimages might have to be stored by a hash of the `[deps]` and `[compat]` sections.
4. It would further complicate the overall CI setup.
### Further procedure
We first need to figure out if we think this is possible from an a priori analysis, and decide that the effort could be worth it. Then we'd need to time our current test setups to figure out whether this actually would speed things up by testing it manually on a local workstation. Finally, we'd need someone to do it :-)
Contributor guide
Assessment
This issue has not been assessed yet.