llvm / llvm/llvm-project

[lit] Scalene profiling of check-llvm and llvm-mca

Open
#218,095 0 comments 0 reactions 0 assignees View on GitHub
tools:llvm-lit
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

### check-llvm profile

Ran `check-llvm` (76,966 discovered tests, `-j10`, macOS/arm64) under Scalene with `--use-virtual-time --profile-all --cpu-percent-threshold 0`.

Aggregate CPU time across every sampled line in every profiled file:

| | % |
|---|---|
| Python | 0.92 |
| Native/C | 8.72 |
| System (blocked in syscalls/I/O) | 77.08 |

Per-file share of CPU time:

| file | % |
|---|---|
| `threading.py` (stdlib) | 60.58 |
| `multiprocessing/connection.py` (stdlib) | 10.46 |
| `lit/TestRunner.py` | 7.96 |
| `lit/run.py` | 7.63 |
| `lit/formats/base.py` | 0.05 |
| `lit/formats/googletest.py` | 0.03 |
| `lit/TestingConfig.py` | 0.01 |

The two dominant lines are both inside `ProcessPoolExecutor`'s own management machinery, not lit's application code:

```
threading.py:870 self._target(*self._args, **self._kwargs) 6.86% C / 53.71% sys
connection.py:368 n = write(self._handle, buf) 1.00% C / 8.91% sys
```

`threading.py:870` is the manager thread's target function call, the time spent blocked waiting inside the executor's own bookkeeping loop. `connection.py:368` is the pickled-result write back through the IPC pipe. Together they account for roughly 71% of all CPU time in the run.

lit's own files (`TestRunner.py` + `run.py` + formats) combine for under 16% of total CPU time. Within those files no single line clears 1%, except the substitution regex sub in `TestRunner.py:1770` (3.54% sys) and the completion-queue poll in `run.py:292` (6.77% sys, `completed.get(timeout=...)`).

At full-suite scale, lit's own Python is not where the time goes. The `ProcessPoolExecutor` management thread and the IPC transport moving results back to the main process are the larger cost.

### check-llvm vs llvm-mca

Both runs used identical flags (`--use-virtual-time --profile-all --cpu-percent-threshold 0`, `-j10`). `llvm-mca` is small enough (~1.8K tests) that per-test Python overhead shows up as distinct, individually-visible lines. On `check-llvm` (~77K tests) those same costs are still present, but a much larger share of total time goes into the executor's own machinery instead.

**Aggregate CPU split:**

| | llvm-mca | check-llvm |
|---|---:|---:|
| Python | 0.62% | 0.92% |
| Native/C | 7.06% | 8.72% |
| System (blocked) | 85.57% | 77.08% |

**Per-file share of CPU time:**

| file | llvm-mca | check-llvm |
|---|---:|---:|
| `threading.py` (stdlib) | 66.67% | 60.58% |
| `lit/TestRunner.py` | 10.35% | 7.96% |
| `lit/run.py` | 1.66% | 7.63% |
| `multiprocessing/connection.py` (stdlib) | 2.07% | 10.46% |
| `lit/ShUtil.py` | 1.94% | — (not in top files) |
| `pathlib.py` (stdlib) | 3.10% | — |
| `subprocess.py` (stdlib) | 2.70% | — |
| `re.py` (stdlib) | 2.45% | — |
| `psutil/_psosx.py` | 2.30% | — |

The comparision shows two differences:

1. **`multiprocessing/connection.py` (the pipe write-back of pickled results) grows from 2.07% to 10.46%**, going from 1.8K to 77K tests, a ~5x jump in share for a ~43x jump in test count. `run.py` also grows, from 1.66% to 7.63%. This suggests dispatch and result-collection become more significant as the test count increases. That's separate from `threading.py`, whose share stays roughly constant across both suites.
2. **`pathlib.py`, `subprocess.py`, `re.py`, and `psutil` only clear visibility on the small suite.**
These costs are associated with per-test setup, including directory creation, environment construction, regex work, and process inspection. They're large enough relative to `llvm-mca`'s sub-millisecond tests to show up individually, and small enough relative to `check-llvm`'s `llc`/`clang` invocations to disappear into the aggregate.

`threading.py:870` (`self._target(...)`, the executor management thread) is the single largest line in both profiles: 60.85% sys on llvm-mca, 53.71% sys on check-llvm. The remaining lines are substantially smaller.

```
llvm-mca, top TestRunner.py lines:
line py% c% sys% code
1770 0.06 0.11 2.11 ln = _caching_re_compile(a).sub(str(b), escapePercents(ln))
1328 0.04 0.07 1.36 f.close()
1761 0.04 0.08 1.25 for a, b in substitutions:
1308 0.02 0.05 0.85 for match in keywords_re.finditer(data):
```

CC: @boomanaiden154 @ilovepi @petrhosek

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the Scalene profiles for check-llvm and llvm-mca with the stated flags, then read lit/TestRunner.py and lit/run.py alongside the ProcessPoolExecutor management and multiprocessing/connection.py paths mentioned in the report. The issue does not define a concrete change or completion condition, so an actionable optimization target and benchmark would need to be established first.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.