pickle.load() significantly slows down with each call on Windows11 but not on Linux

Bug report

Bug description:

My task is to read millions of numpy images and get a region dynamically. The application dictates that the images are stored in batches of about 3000 to 6000 in files. These files contain a pickled dict of numpy arrays. On Windows, reading gets dramatically slower with each call, see logs below. These logs are run on two laptops with completely identical hardware, including 40GB of RAM. While Windows is much slower anyways, it should not get slower with each call?

import os
import pickle
import time
import platform
import sys

import numpy as np
import psutil

filePath = r'C:\images.pkl'

print(f"Versions: {platform.system()=}, {platform.release()=}, {platform.version()=}, {sys.version=}, {np.__version__=}")

imagesDict = {i: np.random.randint(0, 255, (300, 300), dtype=np.uint8) for i in range(4000)}
with open(filePath, 'wb') as file:
    pickle.dump(imagesDict, file, pickle.HIGHEST_PROTOCOL)


thumbs = []
num_image_sets = 0
durations_s_sum = 0.
for i in range(500):
    start_s = time.perf_counter()
    with open(filePath, 'rb') as file:
        imagesDict: dict[int, np.ndarray] = pickle.load(file)
        for key in imagesDict.keys():
            image = imagesDict[key]
            thumb = image[:50, :50].copy()
            thumbs.append(thumb)

    durations_s_sum += (time.perf_counter() - start_s)
    num_image_sets += 1
    if 50 <= num_image_sets:
        memory_info = psutil.Process(os.getpid()).memory_info()
        print(f"{durations_s_sum:4.1f}s for 50 pickle files, rss={memory_info.rss/1024/1024:6,.0f}MB, vms={memory_info.vms/1024/1024:6,.0f}MB")
        durations_s_sum = 0.
        num_image_sets = 0

Windows 11

# Versions: platform.system()='Windows', platform.release()='10', platform.version()='10.0.22631', sys.version='3.11.5 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:26:23) [MSC v.1916 64 bit (AMD64)]', np.__version__='1.26.3'
# 11.7s for 50 pickle files, rss= 1,211MB, vms= 1,215MB
# 11.8s for 50 pickle files, rss= 1,492MB, vms= 1,499MB
# 13.8s for 50 pickle files, rss= 2,272MB, vms= 2,302MB
# 15.7s for 50 pickle files, rss= 2,802MB, vms= 2,845MB
# 18.3s for 50 pickle files, rss= 3,328MB, vms= 3,383MB
# 21.0s for 50 pickle files, rss= 3,837MB, vms= 3,905MB
# 25.6s for 50 pickle files, rss= 4,369MB, vms= 4,448MB
# 28.0s for 50 pickle files, rss= 4,898MB, vms= 4,989MB
# 32.3s for 50 pickle files, rss= 5,427MB, vms= 5,530MB
# 36.7s for 50 pickle files, rss= 5,966MB, vms= 6,081MB

Linux

# Versions: platform.system()='Linux', platform.release()='6.0.12-76060012-generic', platform.version()='#202212290932~1674066459~20.04~3cd2bf3-Ubuntu SMP PREEMPT_DYNAMI', sys.version='3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0]', np.__version__='1.23.5'
#  2.8s for 50 pickle files, rss= 1,222MB, vms= 2,229MB
#  2.8s for 50 pickle files, rss= 1,730MB, vms= 2,738MB
#  2.8s for 50 pickle files, rss= 2,238MB, vms= 3,246MB
#  2.8s for 50 pickle files, rss= 2,747MB, vms= 3,754MB
#  2.8s for 50 pickle files, rss= 3,255MB, vms= 4,263MB
#  2.8s for 50 pickle files, rss= 3,764MB, vms= 4,772MB
#  2.8s for 50 pickle files, rss= 4,272MB, vms= 5,280MB
#  2.8s for 50 pickle files, rss= 4,780MB, vms= 5,789MB
#  2.9s for 50 pickle files, rss= 5,288MB, vms= 6,298MB
#  2.9s for 50 pickle files, rss= 5,797MB, vms= 6,806MB

CPython versions tested on:

3.11

Operating systems tested on:

Windows

Das Issue nennt keine Repository-Dateien oder Tests; beginne damit, den bereitgestellten pickle.load()-Reproducer unter Windows und Linux mit den gemeldeten Versionen auszuführen und dabei Zeit- und Speicherverbrauch zu überwachen. Erledigt ist die Aufgabe, wenn die fortschreitende Verlangsamung unter Windows erklärt und mit Regressionstests abgedeckt wurde.