rust-lang / rust-lang/libz-sys

Very slow compilation under Windows with libz-ng-sys

Open
#171 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

question
Dominant language
Rust
Stars
138
Forks
88
PR merge metrics
No merged PRs in 30d

Description

I wasn't quite sure if I should make the issue here or on the flate2 repo, but here we go:

I noticed for a long time that the longest crate to compile on one of my projects is libz-ng-sys that I use through flate2.
I do a lot of testing with different configurations for fun, so I end up cargo cleaning and then rebuilding a lot, so it's particularly bad.

I use the libz backend because the pure rust one is significantly slower, and the problem is even worse on debug.
With a debug full build on said project, on my machine:
windows, libz backend: ~33s
windows, rust backend: ~10s
wsl linux, libz backend: ~14s

Now, I did a bit of investigation on a separate project that only includes libz-ng-sys, by running both commands manually on an empty folder that the build script usually executes, and:

Creating the initial project

This takes ~19s on my system. A big part of it seems to be checking for unixy headers, and the other big part is from checking simd intrinsics. Most of the headers and features are never found for me. This is a lot of time just for creating a project.

Compiling the sources

This takes ~5s on my system. It is not outrageous compared to the first step, but the compilation is single threaded even if the script tries to pass "--parallel" "20". Unfortunately, that's not enough to enable parallel compilation on MSBuild, and it actually actively hurts it. See this SO thread.

Actually passing the multi processing flags correctly brings this down to ~2s, which is better.

Solutions?

The second step can be fixed fairly easily I think, but the first step is a problem so much bigger than the second.
Something in the Windows ecosystem really doesn't play well with CMake, given that the build under Linux is far more reasonable.

Maybe the CMake can be optimized? Maybe ditching CMake completely and just calling cl directly with all the right options? Maybe clang-cl does a better job and we should suggest it to people? Or if the problem is MSBuild, then also have an option for Ninja or something else?

In conclusion, I don't know.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the Windows build timings for libz-ng-sys, including project creation and source compilation through the build script. Compare the CMake/MSBuild path with the mentioned cl, clang-cl, and Ninja options, and inspect how the script passes parallel flags. Done means a concrete, measured reduction in Windows build time without regressing other platforms.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, rust
Domain
build-system, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.