Improving the daily stubtest check for third-party stubs
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5.1k
- Forks
- 2.1k
- Avg merge
- 1d 19h
- Merged PRs (30d)
- 82
Description
Currently, stubtest checks the stdlib and all third-party stubs daily and will create a new issue if there are discrepancies. While this is fine for the stdlib, it gets annoying for third-party stubs, and doesn't scale well. We either need to fix third-party stubs to a fixed version, which is often overly specific and causes a new stubsabot issue whenever a new micro version is released, or we need to use a range, which has the danger of triggering a new issue daily until fixed. It also means that third-party stubs get checked every night, no matter whether a new package was released or not.
Instead I suggest we extend stubsabot to run on every newly released package version. If the new release is outside the supported version range of the stubs, stubsabot will function as it does now. If the new release is within the supported version range, stubsabot will run stubtest against the new package and open an issue specific for that package if stubtest fails.
This would make fixing these issues less pressing, but also enable us to use more relaxed version ranges. Changes in the "most annoying" packages often only concern internal API and/or are backwards compatible. In the future, we might even consider opening up the ranges more, for example by using >= X.Y.Z, < (X-1) for packages following semver.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the stubsabot implementation and the daily stubtest check, then trace how package versions and supported ranges are discovered. Review the existing issue-creation path and package-release integration. Done means new releases trigger checks for affected third-party stubs, while in-range failures open package-specific issues without nightly repetition.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100