bazel-contrib / bazel-contrib/rules_python

venv site-packages building requires millions of actions

未关闭
#3,401 7 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Starlark
星标
688
派生
721
平均合并
15 小时 7 分钟
30 天内合并 PR
76

描述

On slack, a user reported that using the venv_site_packages=yes flag caused millions of actions to occur. More user details would help verify its due to the flag.

That said but millions of actions is believable if there are many large targets and the conflict detection ends up having to recreate every file under site-packages. e.g. Given 100 tests (reasonable number), and 50,000 files in their deps (large, but not unheard of), then you end up with 5 million actions. No matter what, bazel is going to have to materialize 5 million files in such a case, but if we have to, at the analysis phase, perform 5M symlink() re-declarations, that's a decent amount of overhead.

At the least, we need to optimize the conflict resolution logic. When I fixed it to handle shared libraries better, I just did a simple and naive implementation. A more optimized version would look for a new minimal set of directories/files to symlink instead of just symlinking all of them. Similarly, runfiles.root_symlinks might come in handy here to avoid most symlink()-to-File action calls.

From slack:

>
> Kris Wilson
> Oct 19th at 8:55 PM
> this also seems to cause the number of actions for a bazel test //... to shoot up into the multi-millions.. which is strange. is this creating one action per symlinked file in the venv? (edited)

贡献指南

打开贡献指南

调研方向

首先使用 venv_site_packages=yes 重现一个大型测试图,并检查创建 symlink 操作的冲突解决逻辑。比较优化前后的操作数量和行为;在保留软件包解析的同时避免数百万个冗余文件或 symlink 操作,即表示完成。

由索引模型根据 Issue 内容生成。

评估

技术栈
python
领域
build-system
Issue 类型
缺陷
难度
4/5
预计耗时
3-5 天
活跃度
冷清
描述清晰度
基本清楚
新手友好度
42/100

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。