microsoft / microsoft/pyright

Whole-project analysis drops real errors at low thread counts (per-file and higher --threads recover them); run-to-run unstable (related to #9642)

Open
#11,536 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
15.6k
Forks
1.8k
Avg merge
12h 13m
Merged PRs (30d)
52

Description

## Summary

At the default thread count (and with `--threads 1`), the whole-project run silently drops real errors that per-file analysis reports. Raising the thread count (`--threads 8`) recovers them and also removes the run-to-run variation. In this project the default whole-project run on macOS misses 2 real `reportArgumentType` errors (both on values that can be `None`) that per-file analysis, the Linux default run, and `--threads 8` on either OS all report. This looks like the workspace-vs-per-file effect in #9642, driven by how files are distributed across worker threads.

## Environment

- pyright **1.1.411** (basedpyright 1.39.9 reproduces the same behavior)
- Python **3.14.3**, CPython from python-build-standalone via `uv`, on both platforms
- macOS **arm64** (Darwin) and Linux **aarch64** (Debian, Docker on the same Apple Silicon host, both see 18 cores)
- `typeCheckingMode: standard`, `include: ["."]`, standard `exclude` (dotfiles, node_modules, `__pycache__`, tests, migrations, typings). No `extraPaths` or `ignore`.

## Project scale

- Files analyzed by pyright (`summary.filesAnalyzed`): **3,423**
- Tracked `.py` files: **5,261** total (about 3,521 outside tests, migrations, test_data, typings)
- Python LOC: **about 693k** total (about 351k outside those directories)
- Third-party dependencies resolved from the lockfile: **269** packages (heavy typed deps such as sqlalchemy, pandas, pydantic, botocore, lxml, scipy, pyarrow)
- Whole-project diagnostics (standard mode): **about 11,140 errors, about 115 warnings**. The exact total varies by thread count, OS, and run, which is the subject of this report.
- Analysis time: **about 22s** locally (macOS arm64), **about 3 min** in CI

## The errors are real

Two `reportArgumentType` diagnostics, each passing a value that can be `None` into a parameter typed non-`None` (one via a helper that returns `T | None`, one via an enum lookup that returns `None` when no match is found). Both are true positives and should always be reported. (Names anonymized; the source is proprietary.)

## Finding 1: thread count governs how many real errors are detected

Counts for those two errors on the whole project (no baseline in play):

| configuration | macOS arm64 | Linux aarch64 |
|---|---|---|
| single file (`pyright file.py`) | both reported | both reported |
| whole project, default | both **missing**, total 11140 | one reported, the other flickers |
| whole project, `--threads 1` | both **missing** | same as default |
| whole project, `--threads 8` | both reported, total 11142, stable | both reported, total 11141, stable |

Per-file analysis reports both errors on both platforms. The whole-project run at the default (and `--threads 1`) under-reports relative to per-file analysis and to `--threads 8`. Raising the thread count recovers the dropped errors. macOS at the default is the worst case: it drops both errors, while the Linux default keeps one and flickers on the other.

## Finding 2: run-to-run instability at the default thread count (Linux)

Repeated identical whole-project runs at the default thread count. Signature is the md5 of the sorted `(file, line, rule)` set:

| run | total errors | set signature |
|---|---|---|
| 1 | 11137 | `ead3dc75` |
| 2 | 11137 | `ead3dc75` |
| 3 | 11137 | `ead3dc75` |
| 4 | 11138 | `ceb2718c` |

The set changes between otherwise-identical runs (run 4 reports one diagnostic that runs 1 to 3 do not). The flickering diagnostic is itself one of the two real errors above, so this is a genuine error being intermittently missed. At `--threads 8` this run-to-run variation was gone in our runs (both platforms produced a stable set across repeated runs).

In a separate case the same diagnostic appears on every run, but the inferred type shown in its message alternates across runs between `T` and `Self@T` (a base method's return type). So the instability is in the inference itself, beyond whether a diagnostic appears.

## Residual difference at --threads 8

At `--threads 8` the only remaining macOS-vs-Linux difference is a `reportMissingImports` on a private CPython stdlib module that the code imports directly (`from _pydatetime import date`). pyright resolves it on Linux but not on macOS. That is import resolution, separate from the type-inference behavior above.

## What I ruled out

- **Python patch version**: controlled at 3.14.3 on both platforms.
- **CPU architecture**: aarch64 on both sides.
- Thread count is not ruled out. It is central, see Finding 1.

## Relation to #9642

This looks like the same underlying issue as #9642 (whole-workspace analysis missing errors that per-file analysis finds, with run-to-run variability tied to file distribution across threads). This report adds concrete real errors (false negatives, not just count differences) that the whole-project run drops at the default thread count, shows that `--threads 8` recovers them and stabilizes the run-to-run set, and notes that macOS at the default is worse than Linux.

## Why there is no minimal repro

The dropped errors are themselves minimal (a single file reproduces each one). The dropping is a whole-project-scale effect, and the project is proprietary. I am happy to help reproduce: bisect, run controlled experiments, test a build, or share details privately.

## Expected behavior

Whole-project analysis should report the same errors as per-file analysis, regardless of thread count, and be stable across runs. A genuine `reportArgumentType` should never be silently dropped.

Contributor guide

Open the contributing guide

Research direction

Start with whole-project analysis and worker-thread distribution, comparing per-file runs with whole-project runs at the default, --threads 1, and --threads 8. Reproduce the missing reportArgumentType diagnostics and run-to-run variation if possible; done means whole-project results match per-file analysis and remain stable across thread counts and repeated runs.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
compilers, devtools
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.