bazelbuild / bazelbuild/rules_pkg

pkg_zip is very very slow with rules_python

Open
#795 7 comments 1 reaction 0 assignees View on GitHub
P3
Dominant language
Starlark
Stars
253
Forks
221
Avg merge
10h 33m
Merged PRs (30d)
1

Description

There are significant performance issues with pkg_zip in a bazel environment (on Windows). We were finding when bazel was simultaneously packaging many zips, each one could take 45s instead of 2s expected. We discovered that this is because Bazel's hermetic python toolchain (rules_python) uncompresses many files to prepare for every single python invocation. More discussion here: https://bazelbuild.slack.com/archives/CA306CEV6/p1701253691249489

There appear to be two workarounds:
- use a local python instead of rules_python
- don't use python

I've done a hacky port of build_zip tool to Go using Github Copilot, and it resolves our issue. I'm sharing it here in the hopes that it might be adopted by rules_pkg as an optional alternate language for those facing the same issue.

https://github.com/peakschris/build_zip_go

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.