python / python/typeshed

Improving the daily stubtest check for third-party stubs

Open
#14,838 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

project: infrastructure
Dominant language
Python
Stars
5.1k
Forks
2.1k
Avg merge
1d 19h
Merged PRs (30d)
82

Description

Currently, stubtest checks the stdlib and all third-party stubs daily and will create a new issue if there are discrepancies. While this is fine for the stdlib, it gets annoying for third-party stubs, and doesn't scale well. We either need to fix third-party stubs to a fixed version, which is often overly specific and causes a new stubsabot issue whenever a new micro version is released, or we need to use a range, which has the danger of triggering a new issue daily until fixed. It also means that third-party stubs get checked every night, no matter whether a new package was released or not.

Instead I suggest we extend stubsabot to run on every newly released package version. If the new release is outside the supported version range of the stubs, stubsabot will function as it does now. If the new release is within the supported version range, stubsabot will run stubtest against the new package and open an issue specific for that package if stubtest fails.

This would make fixing these issues less pressing, but also enable us to use more relaxed version ranges. Changes in the "most annoying" packages often only concern internal API and/or are backwards compatible. In the future, we might even consider opening up the ranges more, for example by using >= X.Y.Z, < (X-1) for packages following semver.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the stubsabot implementation and the daily stubtest check, then trace how package versions and supported ranges are discovered. Review the existing issue-creation path and package-release integration. Done means new releases trigger checks for affected third-party stubs, while in-range failures open package-specific issues without nightly repetition.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.