[lit] Scalene profiling of check-llvm and llvm-mca
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
### check-llvm profile
Ran `check-llvm` (76,966 discovered tests, `-j10`, macOS/arm64) under Scalene with `--use-virtual-time --profile-all --cpu-percent-threshold 0`.
Aggregate CPU time across every sampled line in every profiled file:
| | % |
|---|---|
| Python | 0.92 |
| Native/C | 8.72 |
| System (blocked in syscalls/I/O) | 77.08 |
Per-file share of CPU time:
| file | % |
|---|---|
| `threading.py` (stdlib) | 60.58 |
| `multiprocessing/connection.py` (stdlib) | 10.46 |
| `lit/TestRunner.py` | 7.96 |
| `lit/run.py` | 7.63 |
| `lit/formats/base.py` | 0.05 |
| `lit/formats/googletest.py` | 0.03 |
| `lit/TestingConfig.py` | 0.01 |
The two dominant lines are both inside `ProcessPoolExecutor`'s own management machinery, not lit's application code:
```
threading.py:870 self._target(*self._args, **self._kwargs) 6.86% C / 53.71% sys
connection.py:368 n = write(self._handle, buf) 1.00% C / 8.91% sys
```
`threading.py:870` is the manager thread's target function call, the time spent blocked waiting inside the executor's own bookkeeping loop. `connection.py:368` is the pickled-result write back through the IPC pipe. Together they account for roughly 71% of all CPU time in the run.
lit's own files (`TestRunner.py` + `run.py` + formats) combine for under 16% of total CPU time. Within those files no single line clears 1%, except the substitution regex sub in `TestRunner.py:1770` (3.54% sys) and the completion-queue poll in `run.py:292` (6.77% sys, `completed.get(timeout=...)`).
At full-suite scale, lit's own Python is not where the time goes. The `ProcessPoolExecutor` management thread and the IPC transport moving results back to the main process are the larger cost.
### check-llvm vs llvm-mca
Both runs used identical flags (`--use-virtual-time --profile-all --cpu-percent-threshold 0`, `-j10`). `llvm-mca` is small enough (~1.8K tests) that per-test Python overhead shows up as distinct, individually-visible lines. On `check-llvm` (~77K tests) those same costs are still present, but a much larger share of total time goes into the executor's own machinery instead.
**Aggregate CPU split:**
| | llvm-mca | check-llvm |
|---|---:|---:|
| Python | 0.62% | 0.92% |
| Native/C | 7.06% | 8.72% |
| System (blocked) | 85.57% | 77.08% |
**Per-file share of CPU time:**
| file | llvm-mca | check-llvm |
|---|---:|---:|
| `threading.py` (stdlib) | 66.67% | 60.58% |
| `lit/TestRunner.py` | 10.35% | 7.96% |
| `lit/run.py` | 1.66% | 7.63% |
| `multiprocessing/connection.py` (stdlib) | 2.07% | 10.46% |
| `lit/ShUtil.py` | 1.94% | — (not in top files) |
| `pathlib.py` (stdlib) | 3.10% | — |
| `subprocess.py` (stdlib) | 2.70% | — |
| `re.py` (stdlib) | 2.45% | — |
| `psutil/_psosx.py` | 2.30% | — |
The comparision shows two differences:
1. **`multiprocessing/connection.py` (the pipe write-back of pickled results) grows from 2.07% to 10.46%**, going from 1.8K to 77K tests, a ~5x jump in share for a ~43x jump in test count. `run.py` also grows, from 1.66% to 7.63%. This suggests dispatch and result-collection become more significant as the test count increases. That's separate from `threading.py`, whose share stays roughly constant across both suites.
2. **`pathlib.py`, `subprocess.py`, `re.py`, and `psutil` only clear visibility on the small suite.**
These costs are associated with per-test setup, including directory creation, environment construction, regex work, and process inspection. They're large enough relative to `llvm-mca`'s sub-millisecond tests to show up individually, and small enough relative to `check-llvm`'s `llc`/`clang` invocations to disappear into the aggregate.
`threading.py:870` (`self._target(...)`, the executor management thread) is the single largest line in both profiles: 60.85% sys on llvm-mca, 53.71% sys on check-llvm. The remaining lines are substantially smaller.
```
llvm-mca, top TestRunner.py lines:
line py% c% sys% code
1770 0.06 0.11 2.11 ln = _caching_re_compile(a).sub(str(b), escapePercents(ln))
1328 0.04 0.07 1.36 f.close()
1761 0.04 0.08 1.25 for a, b in substitutions:
1308 0.02 0.05 0.85 for match in keywords_re.finditer(data):
```
CC: @boomanaiden154 @ilovepi @petrhosek
Contributor guide
Research direction
Start by reproducing the Scalene profiles for check-llvm and llvm-mca with the stated flags, then read lit/TestRunner.py and lit/run.py alongside the ProcessPoolExecutor management and multiprocessing/connection.py paths mentioned in the report. The issue does not define a concrete change or completion condition, so an actionable optimization target and benchmark would need to be established first.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100