Whole-project analysis drops real errors at low thread counts (per-file and higher --threads recover them); run-to-run unstable (related to #9642)
- Dominant language
- Python
- Stars
- 15.6k
- Forks
- 1.8k
- Avg merge
- 12h 13m
- Merged PRs (30d)
- 52
Description
## Summary
At the default thread count (and with `--threads 1`), the whole-project run silently drops real errors that per-file analysis reports. Raising the thread count (`--threads 8`) recovers them and also removes the run-to-run variation. In this project the default whole-project run on macOS misses 2 real `reportArgumentType` errors (both on values that can be `None`) that per-file analysis, the Linux default run, and `--threads 8` on either OS all report. This looks like the workspace-vs-per-file effect in #9642, driven by how files are distributed across worker threads.
## Environment
- pyright **1.1.411** (basedpyright 1.39.9 reproduces the same behavior)
- Python **3.14.3**, CPython from python-build-standalone via `uv`, on both platforms
- macOS **arm64** (Darwin) and Linux **aarch64** (Debian, Docker on the same Apple Silicon host, both see 18 cores)
- `typeCheckingMode: standard`, `include: ["."]`, standard `exclude` (dotfiles, node_modules, `__pycache__`, tests, migrations, typings). No `extraPaths` or `ignore`.
## Project scale
- Files analyzed by pyright (`summary.filesAnalyzed`): **3,423**
- Tracked `.py` files: **5,261** total (about 3,521 outside tests, migrations, test_data, typings)
- Python LOC: **about 693k** total (about 351k outside those directories)
- Third-party dependencies resolved from the lockfile: **269** packages (heavy typed deps such as sqlalchemy, pandas, pydantic, botocore, lxml, scipy, pyarrow)
- Whole-project diagnostics (standard mode): **about 11,140 errors, about 115 warnings**. The exact total varies by thread count, OS, and run, which is the subject of this report.
- Analysis time: **about 22s** locally (macOS arm64), **about 3 min** in CI
## The errors are real
Two `reportArgumentType` diagnostics, each passing a value that can be `None` into a parameter typed non-`None` (one via a helper that returns `T | None`, one via an enum lookup that returns `None` when no match is found). Both are true positives and should always be reported. (Names anonymized; the source is proprietary.)
## Finding 1: thread count governs how many real errors are detected
Counts for those two errors on the whole project (no baseline in play):
| configuration | macOS arm64 | Linux aarch64 |
|---|---|---|
| single file (`pyright file.py`) | both reported | both reported |
| whole project, default | both **missing**, total 11140 | one reported, the other flickers |
| whole project, `--threads 1` | both **missing** | same as default |
| whole project, `--threads 8` | both reported, total 11142, stable | both reported, total 11141, stable |
Per-file analysis reports both errors on both platforms. The whole-project run at the default (and `--threads 1`) under-reports relative to per-file analysis and to `--threads 8`. Raising the thread count recovers the dropped errors. macOS at the default is the worst case: it drops both errors, while the Linux default keeps one and flickers on the other.
## Finding 2: run-to-run instability at the default thread count (Linux)
Repeated identical whole-project runs at the default thread count. Signature is the md5 of the sorted `(file, line, rule)` set:
| run | total errors | set signature |
|---|---|---|
| 1 | 11137 | `ead3dc75` |
| 2 | 11137 | `ead3dc75` |
| 3 | 11137 | `ead3dc75` |
| 4 | 11138 | `ceb2718c` |
The set changes between otherwise-identical runs (run 4 reports one diagnostic that runs 1 to 3 do not). The flickering diagnostic is itself one of the two real errors above, so this is a genuine error being intermittently missed. At `--threads 8` this run-to-run variation was gone in our runs (both platforms produced a stable set across repeated runs).
In a separate case the same diagnostic appears on every run, but the inferred type shown in its message alternates across runs between `T` and `Self@T` (a base method's return type). So the instability is in the inference itself, beyond whether a diagnostic appears.
## Residual difference at --threads 8
At `--threads 8` the only remaining macOS-vs-Linux difference is a `reportMissingImports` on a private CPython stdlib module that the code imports directly (`from _pydatetime import date`). pyright resolves it on Linux but not on macOS. That is import resolution, separate from the type-inference behavior above.
## What I ruled out
- **Python patch version**: controlled at 3.14.3 on both platforms.
- **CPU architecture**: aarch64 on both sides.
- Thread count is not ruled out. It is central, see Finding 1.
## Relation to #9642
This looks like the same underlying issue as #9642 (whole-workspace analysis missing errors that per-file analysis finds, with run-to-run variability tied to file distribution across threads). This report adds concrete real errors (false negatives, not just count differences) that the whole-project run drops at the default thread count, shows that `--threads 8` recovers them and stabilizes the run-to-run set, and notes that macOS at the default is worse than Linux.
## Why there is no minimal repro
The dropped errors are themselves minimal (a single file reproduces each one). The dropping is a whole-project-scale effect, and the project is proprietary. I am happy to help reproduce: bisect, run controlled experiments, test a build, or share details privately.
## Expected behavior
Whole-project analysis should report the same errors as per-file analysis, regardless of thread count, and be stable across runs. A genuine `reportArgumentType` should never be silently dropped.
Contributor guide
Research direction
Start with whole-project analysis and worker-thread distribution, comparing per-file runs with whole-project runs at the default, --threads 1, and --threads 8. Reproduce the missing reportArgumentType diagnostics and run-to-run variation if possible; done means whole-project results match per-file analysis and remain stable across thread counts and repeated runs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- compilers, devtools
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100