pickle.load() significantly slows down with each call on Windows11 but not on Linux

Bug report

Bug description:

My task is to read millions of numpy images and get a region dynamically. The application dictates that the images are stored in batches of about 3000 to 6000 in files. These files contain a pickled dict of numpy arrays. On Windows, reading gets dramatically slower with each call, see logs below. These logs are run on two laptops with completely identical hardware, including 40GB of RAM. While Windows is much slower anyways, it should not get slower with each call?

import os
import pickle
import time
import platform
import sys

import numpy as np
import psutil

filePath = r'C:\images.pkl'

print(f"Versions: {platform.system()=}, {platform.release()=}, {platform.version()=}, {sys.version=}, {np.__version__=}")

imagesDict = {i: np.random.randint(0, 255, (300, 300), dtype=np.uint8) for i in range(4000)}
with open(filePath, 'wb') as file:
    pickle.dump(imagesDict, file, pickle.HIGHEST_PROTOCOL)


thumbs = []
num_image_sets = 0
durations_s_sum = 0.
for i in range(500):
    start_s = time.perf_counter()
    with open(filePath, 'rb') as file:
        imagesDict: dict[int, np.ndarray] = pickle.load(file)
        for key in imagesDict.keys():
            image = imagesDict[key]
            thumb = image[:50, :50].copy()
            thumbs.append(thumb)

    durations_s_sum += (time.perf_counter() - start_s)
    num_image_sets += 1
    if 50 <= num_image_sets:
        memory_info = psutil.Process(os.getpid()).memory_info()
        print(f"{durations_s_sum:4.1f}s for 50 pickle files, rss={memory_info.rss/1024/1024:6,.0f}MB, vms={memory_info.vms/1024/1024:6,.0f}MB")
        durations_s_sum = 0.
        num_image_sets = 0

Windows 11

# Versions: platform.system()='Windows', platform.release()='10', platform.version()='10.0.22631', sys.version='3.11.5 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:26:23) [MSC v.1916 64 bit (AMD64)]', np.__version__='1.26.3'
# 11.7s for 50 pickle files, rss= 1,211MB, vms= 1,215MB
# 11.8s for 50 pickle files, rss= 1,492MB, vms= 1,499MB
# 13.8s for 50 pickle files, rss= 2,272MB, vms= 2,302MB
# 15.7s for 50 pickle files, rss= 2,802MB, vms= 2,845MB
# 18.3s for 50 pickle files, rss= 3,328MB, vms= 3,383MB
# 21.0s for 50 pickle files, rss= 3,837MB, vms= 3,905MB
# 25.6s for 50 pickle files, rss= 4,369MB, vms= 4,448MB
# 28.0s for 50 pickle files, rss= 4,898MB, vms= 4,989MB
# 32.3s for 50 pickle files, rss= 5,427MB, vms= 5,530MB
# 36.7s for 50 pickle files, rss= 5,966MB, vms= 6,081MB

Linux

# Versions: platform.system()='Linux', platform.release()='6.0.12-76060012-generic', platform.version()='#202212290932~1674066459~20.04~3cd2bf3-Ubuntu SMP PREEMPT_DYNAMI', sys.version='3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0]', np.__version__='1.23.5'
#  2.8s for 50 pickle files, rss= 1,222MB, vms= 2,229MB
#  2.8s for 50 pickle files, rss= 1,730MB, vms= 2,738MB
#  2.8s for 50 pickle files, rss= 2,238MB, vms= 3,246MB
#  2.8s for 50 pickle files, rss= 2,747MB, vms= 3,754MB
#  2.8s for 50 pickle files, rss= 3,255MB, vms= 4,263MB
#  2.8s for 50 pickle files, rss= 3,764MB, vms= 4,772MB
#  2.8s for 50 pickle files, rss= 4,272MB, vms= 5,280MB
#  2.8s for 50 pickle files, rss= 4,780MB, vms= 5,789MB
#  2.9s for 50 pickle files, rss= 5,288MB, vms= 6,298MB
#  2.9s for 50 pickle files, rss= 5,797MB, vms= 6,806MB

CPython versions tested on:

3.11

Operating systems tested on:

Windows

この issue ではリポジトリのファイルやテストが指定されていません。まず、報告されたバージョンを使って Windows と Linux で提供された pickle.load() の再現手順を実行し、その間の実行時間とメモリ使用量を監視してください。Windows で段階的に遅くなる原因を説明し、回帰テストでカバーできれば完了です。