llvm / llvm/llvm-project

[LIT] Windows buildbots failed because a worker process crashed

Open
#219,909 3 comments 0 reactions 0 assignees View on GitHub
llvm-lit platform:windows tools:llvm-lit
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

Immediately after #209076 we faced that few Windows buildbots periodically failed because a worker process crashed:
https://lab.llvm.org/buildbot/#/builders/54/builds/21970
https://lab.llvm.org/buildbot/#/builders/155/builds/21998

Note these workers contain few Xeon Gold 6338 and up to 128 logical processors (threads), cmake v4.4.2, python v3.14.7.
Windows Defender is disabled, and the build directory has been added to the exclusions.

Similar hardware with Ubuntu 24.04 and python 3.12.3 seems ok
https://lab.llvm.org/buildbot/#/workers/51

The most frequent issue looks like
https://lab.llvm.org/buildbot/#/builders/54/builds/22839
```
llvm-lit.py: C:\buildbot\as-builder-8\llvm-nvptx-nvidia-win\llvm-project\llvm\utils\lit\lit\main.py:315: warning: Failed to delete temp directory 'c:\buildbot\as-builder-8\llvm-nvptx-nvidia-win\build\lit-tmp-ql1gypry', try upgrading your version of Python to fix this
llvm-lit.py: C:\buildbot\as-builder-8\llvm-nvptx-nvidia-win\llvm-project\llvm\utils\lit\lit\main.py:283: error: a worker process crashed: A process in the process pool was terminated abruptly while the future was running or pending.
```
We can try something like the following around [lit/main.py#L313](https://github.com/llvm/llvm-project/blob/e1a4ef535a07fbd41827f53708b71efc6ad20a68/llvm/utils/lit/lit/main.py#L313)
```
import time

# Inside execute_in_tmp_dir's finally block:
try:
shutil.rmtree(tmp_dir)
except Exception:
time.sleep(1) # Give Windows a moment to release handles
try:
shutil.rmtree(tmp_dir)
except Exception as e:
lit_config.warning("Failed to delete temp directory...")
```
The second issue is the following (no any additional details)
https://lab.llvm.org/buildbot/#/builders/54/builds/22876
```
llvm-lit.py: C:\buildbot\as-builder-8\llvm-nvptx-nvidia-win\llvm-project\llvm\utils\lit\lit\llvm\config.py:64: note: using lit tools: C:\Program Files\Git\usr\bin
llvm-lit.py: C:\buildbot\as-builder-8\llvm-nvptx-nvidia-win\llvm-project\llvm\utils\lit\lit\run.py:168: note: Using 3 pools balancing 128 workers total distributed as [43, 43, 42] (Windows worker limit workaround)
...
llvm-lit.py: C:\buildbot\as-builder-8\llvm-nvptx-nvidia-win\llvm-project\llvm\utils\lit\lit\main.py:283: error: a worker process crashed: A process in the process pool was terminated abruptly while the future was running or pending.
```

Contributor guide

Open the contributing guide

Research direction

Start with llvm/utils/lit/lit/main.py around the execute_in_tmp_dir cleanup at line 313, then inspect the worker-pool handling in llvm/utils/lit/lit/run.py and the Windows configuration in llvm/utils/lit/lit/llvm/config.py. Reproduce or compare the referenced Windows buildbot failures, and consider the issue done when the affected Windows runs no longer report abrupt worker-process termination and the temporary-directory cleanup behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, ci-cd, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.