python / python/cpython

Don't run importlib tests for the "source" version

Open
#92,328 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

tests topic-importlib type-feature
Dominant language
Python
Stars
77.2k
Forks
36k
PR merge metrics
PR metrics pending

Description

Feature or enhancement

Simplify the test suite by not running the importlib tests on the source version, only the frozen one.

Pitch

Currently, we are running the tests against both the source and frozen versions of importlib.

Example:
https://github.com/python/cpython/blob/main/Lib/test/test_importlib/test_api.py#L97-L99

In #92265, we ran into a non-trivial issue where the frozen finders were leaking into source version of the tests, and I suspect some other tests might have the same issue. Me and @ericsnowcurrently spent quite a bit of time tracking it down, though I acknowledge it could have been easier for someone like Brett, who is more versed in this part of the code.

AFAICT, the only version of the modules users will actually use is the frozen one, making me question if the increased complexity in the test suite is worth it. I am under the impression that this was not always the case, which would explain a bit better why we are testing both versions.

So, would it be reasonable to simply remove the source version of the tests? What value is it actually providing? Did I overlook something here?

cc @brettcannon @ericsnowcurrently @warsaw

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the importlib test setup in Lib/test/test_importlib/test_api.py around lines 97-99, then review the source and frozen test paths and the issue described in #92265. Determine what coverage the source-version tests provide; done means documenting the rationale and making the test-suite change only if that coverage is unnecessary.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.