python / python/mypy

Create tool that can infer missing test fixtures

Open
#15,119 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-developer
Dominant language
Python
Stars
20.6k
Forks
3.3k
PR merge metrics
PR metrics pending

Description

Coming up with test fixtures (e.g. [builtins ...]) adds a lot of friction, especially for new contributors. Write a tool to automatically infer and add a missing fixture to newly added test cases. It doesn't need to work perfectly, but we can perhaps implement something that works for, say, 80-90% of test cases.

The tool could work like this:

  • Use git commands to determine newly added test cases by comparing against an ancestor master branch commit.
  • For each new test case that doesn't have a fixture yet, look at the code being tested and guess a suitable fixture. For example, if a test case uses tuples, suggest fixtures/tuple.pyi.

The inference might need some pretty ad hoc heuristics (e.g. which fixture to pick if multiple features are used), but that's okay.

This would need to be documented prominently in the developer docs.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files or entry points are named. Start by locating the existing test-case and fixture conventions, then inspect how developer documentation is organized and how git identifies changes from an ancestor master commit. Done means a documented tool can detect newly added tests without fixtures, suggest fixtures from the tested code, and handle the expected common cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
developer-experience, documentation, testing-qa, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.