Options for improving precompile time

Summary

The current design of Sienna causes large precompilation latency. The modeling flexibility offered by Sienna and JuMP means that the packages are not type stable, and the multi-repo design of Sienna works against the package-local compilation model of Julia. In addition, the wide variety of input data types, custom component types, and different analyses that the user can undertake means that the API surface is very broad.

There are two main recommendations:

1. Create a single meta-package Sienna.jl that wraps and re-exports all other packages. This would enable the use of PrecompileTools.jl inside Sienna.jl, but it requires yet-another major API change, and it goes against the spirit of providing a suite of composable pieces that users can link together as needed. It may also be premature to do so while the ecosystem is still being actively split into different packages.

2. Recommend that users can create a custom sysimage, based on their own specific modeling script, to reduce compilation latency for a given model at the cost of needing to launch Julia with a custom command.

Background

Precompilation latency stems from Julia’s Just-in-Time (JIT) compilation model. When a function is called twice on the same input types, the first time is noticeably slower than the second. The extra time associated with the first call, referred to as precompilation latency, is because Julia’s compiler must run and generate machine code for the particular function call. When the user calls the function a second time, the cached machine code is used instead of compiling a new method. Because the compilation latency happens the first time a function is called in a Julia session, it is referred to colloquially as the time-to-first-X problem.

Precompilation latency is a bad developer experience. Even though Sienna can build and solve large models (relatively) quickly, precompilation latency gives the initial impression of sluggishness. This is particularly critical when on-boarding new users, and for users developing small models. Moreover, most Julia sessions using Sienna build and solve a single problem. Users don’t have long-running Julia sessions where multiple different Sienna models are built and solved, where the compilation cost can be amortized across many runs.

Precompilation latency is caused by the combination of a large API surface area and functions that are not type stable. Because the compilation model operates on combinations of functions and input types, an API design which has many functions and many input types needs to call the compiler many times. This is a problem in Sienna, which, by design, features many different types of components, and supports many different types of optimization and simulation workflows. Moreover, type instability can cause method invalidations, which causes functions that were previously compiled to need to be recompiled.

Suggestions to reduce precompilation latency

Since precompilation latency affects all Julia packages, the community has developed three packages, SnoopCompile.jl, PrecompileTools.jl, and PackageCompiler.jl, to help developers reduce precompile time. This report investigates each in turn with their applicability to Sienna.

Improve code quality

One way to reduce precompilation latency is to reduce type instabilities and make the code easier for Julia to infer. The main benefit of this is that it reduces method invalidations. See https://juliadebug.github.io/SnoopCompile.jl/dev/tutorials/invalidations/ for a tutorial on method invalidations.

To find method invalidations, run:

using SnoopCompileCore
invs = @snoop_invalidations using PowerSystems, PowerOperationsModels
using SnoopCompile, AbstractTrees
trees = invalidation_trees(invs)

This leads to the first culprit: https://github.com/Sienna-Platform/InfrastructureSystems.jl/issues/524, but there are many more.

Many instances of type instability in Sienna stem from intentional design decisions, such as the ability to have a mix of components in the model. There are many parts of JuMP and MathOptInterface that are similarly not type stable because, for example, the type instability in JuMP’s model.moi_backend allows the underlying solver to be changed at run-time. Thus, improving the code quality in Sienna may lead to some improvements, but it will not fix all the issues.

Use PrecompileTools.jl

PrecompileTools.jl is a package for reducing compilation latency. It works by embedding a small script into the source code of your package. The script is evaluated once when the package is installed, and the compiled code is cached for future sessions.

Many packages related to Sienna use PrecompileTools.jl. Examples are JuMP, PowerModels, and HiGHS.

Here is how PrecompileTools is used PowerModels:

module PowerModels

# ... the source code of PowerModels.jl

import PrecompileTools
PrecompileTools.@setup_workload begin
   logger_config!("error")  # Turn off logging for this precompile block
   case3 = joinpath(dirname(@__DIR__), "test/data/matpower/case3.m")
   case9 = joinpath(dirname(@__DIR__), "test/data/matpower/case9.m")
   PrecompileTools.@compile_workload begin
       for case in [case3, case9]
           data = parse_file(case)
           _ = instantiate_model(data, ACPPowerModel, build_opf)
           _ = instantiate_model(data, ACPPowerModel, build_pf)
           _ = instantiate_model(data, DCPPowerModel, build_opf)
           _ = instantiate_model(data, DCPPowerModel, build_pf)
       end
       _ = compute_ac_pf(case9)
       _ = compute_dc_pf(case9)
   end
   logger_config!("info")   # Re-enable default logging
end

end  # module PowerModels

The code inside the PrecompileTools.@compile_workload begin block is compiled and cached for all future sessions.

There are two main limitations with PrecompileTools.jl:

1. The precompile script can reference only functions and packages that are defined inside the module. In the PowerModels case, it can read data and instantiate models, but it cannot solve OPF cases because doing so requires a solver like Ipopt.jl, which is not a direct dependency of PowerModels.jl

2. We must enumerate the cases that we want to precompile. In the PowerModels case, we chose to instantiate only ACPPowerModel and DCPPowerModel for the OPF and PF cases. More exotic cases like second-order cone relaxations are not precompiled, and will incur latency when first called by the user. There is also a trade-off to consider: every user runs this script on installation, so computationally heavy precompile scripts increase the installation time.

Both of these limitations work against the use of PrecompileTools in Sienna.

First, to simplify maintenance, Sienna was decomposed into a number of separate packages. A common user script includes import statements such as:

import PowerNetworkMatrices as PNM
import PowerOperationsModels as POM
import PowerSystems as PSY

This means that there is no single home for the precompile script to live. It cannot live inside PowerSystems because it requires network data from PowerNetworkMatrices, and vice versa.

Second, unlike PowerModels, in which almost all users wish to solve AC-OPF using the polar form with Ipopt and data from a MATPOWER file, SIenna is designed to support a very large number of components and optimization and simulation problem types. This makes designing a precompile script difficult. There is no Sienna equivalent to solve_ac_opf.

In order for Sienna to benefit from PrecompileTools.jl, we would need to create a single meta-package (named, for example, Sienna.jl) that wraps and re-exports all other packages in the Sienna ecosystem. The downside to this approach is that it requires yet-another major API change, and it goes against the spirit of providing a suite of composable pieces that users can link together as needed.

Pull requests related to this option:

• https://github.com/Sienna-Platform/PowerOperationsModels.jl/pull/204
• https://github.com/Sienna-Platform/PowerOperationsModels.jl/pull/208
• https://github.com/Sienna-Platform/PowerOperationsModels.jl/pull/209
• https://github.com/Sienna-Platform/InfrastructureOptimizationModels.jl/pull/145

Use PackageCompiler.jl

PackageCompiler.jl is a package for working with Julia sysimages. A sysimage is a serialized file of a Julia session, including things like loaded packages, global variables, inferred and compiled code. When Julia is started with a sysimage, the session can be deserialized from disk instead of needing to recompile the serialized code from scratch.

Creating a sysimage is simple. First, you need a runnable Julia script, which includes all necessary dependencies, and which does work similar to the work you want to remove precompilation from. Second, use PackageCompiler.create_sysimage to create a custom sysimage:

import PackageCompiler
PackageCompiler.create_sysimage(;
   sysimage_path = "my_sysimage.so",
   precompile_execution_file = "script.jl",
)

Then, Julia can be started with the sysimage using:

julia --sysimage my_sysimage.so

To test the performance of PackageCompiler, we used the Single-step Problem example from the PowerSimulations.jl documentation. The results show that PackageCompiler eliminates the precompilation latency: without the sysimage, the script takes 65 seconds to run; with the sysimage, the script takes 4 seconds to run.

(base) odow@Mac sienna % time julia --project=. script.jl                             
... output omitted ...
get_objective_value(res) = 2.356823683788018e6
julia --project=. script.jl  64.28s user 0.84s system 99% cpu 1:05.12 total

(base) odow@Mac sienna % time julia --project=. --sysimage my_sysimage.dylib script.jl
... output omitted ...
get_objective_value(res) = 2.356823683788018e6
julia --project=. --sysimage my_sysimage.dylib script.jl  3.58s user 0.17s system 138% cpu 2.709 total

The biggest downside to a sysimage is that it requires a fixed set of package versions. If you add a new package or update any dependencies, you must recompile the sysimage.

Another downside is that compiling the sysimage can take many minutes. The sysimage for my test case took 7 minutes:

julia> import PackageCompiler
julia> @time PackageCompiler.create_sysimage(;
         sysimage_path = "my_sysimage.dylib",
         precompile_execution_file = "script.jl",
      )
# ... output omitted ...
410.695423 seconds (2.68 M allocations: 167.845 MiB, 0.04% gc time, 0.07% compilation time: 37% of which was recompilation)

A third downside is that Julia needs to be started with the custom --sysimage option. This means that a sysimage is not a good option to remove precompilation latency for new users, or for developers actively working on a new model, and who may be adding and removing packages as their needs arise.

The best use-case for a custom sysimage is when the model and package versions are stable over a long period, and when the same model needs to be run repeatedly.

using PowerSystems
using PowerSimulations
using HydroPowerSimulations
using PowerSystemCaseBuilder
using HiGHS # solver
using Dates
sys = build_system(PSISystems, "modified_RTS_GMLC_DA_sys")
template_uc = ProblemTemplate()
set_device_model!(template_uc, Line, StaticBranch)
set_device_model!(template_uc, Transformer2W, StaticBranch)
set_device_model!(template_uc, TapTransformer, StaticBranch)
set_device_model!(template_uc, ThermalStandard, ThermalStandardUnitCommitment)
set_device_model!(template_uc, RenewableDispatch, RenewableFullDispatch)
set_device_model!(template_uc, PowerLoad, StaticPowerLoad)
set_device_model!(template_uc, HydroDispatch, HydroDispatchRunOfRiver)
set_device_model!(template_uc, RenewableNonDispatch, FixedOutput)
set_service_model!(template_uc, VariableReserve{ReserveUp}, RangeReserve)
set_service_model!(template_uc, VariableReserve{ReserveDown}, RangeReserve)
set_network_model!(template_uc, NetworkModel(CopperPlatePowerModel))
solver = optimizer_with_attributes(HiGHS.Optimizer, "mip_rel_gap" => 0.5)
problem = DecisionModel(template_uc, sys; optimizer = solver, horizon = Hour(24))
build!(problem; output_dir = mktempdir())
solve!(problem)
res = OptimizationProblemResults(problem)
@show get_objective_value(res)