python / python/cpython

Windows build importlib performance regression

Open
#99,858 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

topic-importlib type-bug
Dominant language
Python
Stars
77.2k
Forks
36k
Avg merge
1d 9h
Merged PRs (30d)
558

Description

Bug report

I've a file import_models.py:

import time
from ctypes import c_longdouble
from multiprocessing import Value


def import_models_func(_, import_value, exec_value):
    init = time.process_time()
    import some_class
    import_value.value = time.process_time() - init
    exec_value.value = 0


if __name__ == '__main__':
    import_models_func(0, Value(c_longdouble, lock=False), Value(c_longdouble, lock=False))

The some_class.py is:

import time


class SomeClass:
    __slots__ = ('a', 'b', 'c')

    def __init__(self):
        self.a = time.time()
        self.b = self.a / 100
        self.c = self.b ** 2

    B = 2
    for _ in range(10):
        B **= 2

    @property
    def a_val(self):
        return self.a

    @a_val.getter
    def a_val(self):
        return self.a

    @property
    def b_val(self):
        return self.b

    @b_val.getter
    def b_val(self):
        return self.b

    @property
    def c_val(self):
        return self.c

    @c_val.getter
    def c_val(self):
        return self.c

The directory containing both files has no __pycache__. I run python -X importtime import_models.py and receive:

import time:       521 |        521 | some_class

Then I run python -m compileall . and re-run the command above, the output is:

import time:       548 |        548 | some_class

I've tried the sampling with multiprocessing governor, which is run with -B. The governor runs this scenario for 100 times, both mean and median statistics values of the timing measured mostly provide with the same results: the timing without __pycache__ is less than when the source files are pre-compiled. Which is a non-sense keeping in mind what the pre-compiled byte-code cache is built for.

I've searched for Windows-related fixes through the versions 3.9 to 3.12 and found nothing about this regression.

Linux build in a native (non-WSL, non-emulated) environment does not have such an issue (the bytecode-compiled version gets imported faster, up to 10 times, than a raw non-compiled source)

Your environment

  • CPython versions tested on: Python 3.9.13 (tags/v3.9.13:6de2ca5, May 17 2022, 16:36:42) [MSC v.1929 64 bit (AMD64)] on win32
  • Operating system and architecture: Windows 10 Enterprise 21H2 19044.2251

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the Windows 10 timing difference with the provided import_models.py and some_class.py using -X importtime, then compare runs before and after python -m compileall . and with -B. Check the importlib and Windows-specific import paths involved in loading the module. Done means identifying the cause of the slower cached import and documenting or fixing the regression with a verified comparison.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
operating-systems, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.