pickle.load() significantly slows down with each call on Windows11 but not on Linux

Bug report

Bug description:

My task is to read millions of numpy images and get a region dynamically. The application dictates that the images are stored in batches of about 3000 to 6000 in files. These files contain a pickled dict of numpy arrays. On Windows, reading gets dramatically slower with each call, see logs below. These logs are run on two laptops with completely identical hardware, including 40GB of RAM. While Windows is much slower anyways, it should not get slower with each call?

import os
import pickle
import time
import platform
import sys

import numpy as np
import psutil

filePath = r'C:\images.pkl'

print(f"Versions: {platform.system()=}, {platform.release()=}, {platform.version()=}, {sys.version=}, {np.__version__=}")

imagesDict = {i: np.random.randint(0, 255, (300, 300), dtype=np.uint8) for i in range(4000)}
with open(filePath, 'wb') as file:
    pickle.dump(imagesDict, file, pickle.HIGHEST_PROTOCOL)


thumbs = []
num_image_sets = 0
durations_s_sum = 0.
for i in range(500):
    start_s = time.perf_counter()
    with open(filePath, 'rb') as file:
        imagesDict: dict[int, np.ndarray] = pickle.load(file)
        for key in imagesDict.keys():
            image = imagesDict[key]
            thumb = image[:50, :50].copy()
            thumbs.append(thumb)

    durations_s_sum += (time.perf_counter() - start_s)
    num_image_sets += 1
    if 50 <= num_image_sets:
        memory_info = psutil.Process(os.getpid()).memory_info()
        print(f"{durations_s_sum:4.1f}s for 50 pickle files, rss={memory_info.rss/1024/1024:6,.0f}MB, vms={memory_info.vms/1024/1024:6,.0f}MB")
        durations_s_sum = 0.
        num_image_sets = 0

Windows 11

# Versions: platform.system()='Windows', platform.release()='10', platform.version()='10.0.22631', sys.version='3.11.5 | packaged by Anaconda, Inc. | (main, Sep 11 2023, 13:26:23) [MSC v.1916 64 bit (AMD64)]', np.__version__='1.26.3'
# 11.7s for 50 pickle files, rss= 1,211MB, vms= 1,215MB
# 11.8s for 50 pickle files, rss= 1,492MB, vms= 1,499MB
# 13.8s for 50 pickle files, rss= 2,272MB, vms= 2,302MB
# 15.7s for 50 pickle files, rss= 2,802MB, vms= 2,845MB
# 18.3s for 50 pickle files, rss= 3,328MB, vms= 3,383MB
# 21.0s for 50 pickle files, rss= 3,837MB, vms= 3,905MB
# 25.6s for 50 pickle files, rss= 4,369MB, vms= 4,448MB
# 28.0s for 50 pickle files, rss= 4,898MB, vms= 4,989MB
# 32.3s for 50 pickle files, rss= 5,427MB, vms= 5,530MB
# 36.7s for 50 pickle files, rss= 5,966MB, vms= 6,081MB

Linux

# Versions: platform.system()='Linux', platform.release()='6.0.12-76060012-generic', platform.version()='#202212290932~1674066459~20.04~3cd2bf3-Ubuntu SMP PREEMPT_DYNAMI', sys.version='3.10.9 (main, Jan 11 2023, 15:21:40) [GCC 11.2.0]', np.__version__='1.23.5'
#  2.8s for 50 pickle files, rss= 1,222MB, vms= 2,229MB
#  2.8s for 50 pickle files, rss= 1,730MB, vms= 2,738MB
#  2.8s for 50 pickle files, rss= 2,238MB, vms= 3,246MB
#  2.8s for 50 pickle files, rss= 2,747MB, vms= 3,754MB
#  2.8s for 50 pickle files, rss= 3,255MB, vms= 4,263MB
#  2.8s for 50 pickle files, rss= 3,764MB, vms= 4,772MB
#  2.8s for 50 pickle files, rss= 4,272MB, vms= 5,280MB
#  2.8s for 50 pickle files, rss= 4,780MB, vms= 5,789MB
#  2.9s for 50 pickle files, rss= 5,288MB, vms= 6,298MB
#  2.9s for 50 pickle files, rss= 5,797MB, vms= 6,806MB

CPython versions tested on:

3.11

Operating systems tested on:

Windows

Issue không nêu tên tệp hoặc bài kiểm thử nào trong repository; hãy bắt đầu bằng cách chạy reproducer pickle.load() được cung cấp trên Windows và Linux với các phiên bản đã báo cáo, đồng thời theo dõi thời gian và mức sử dụng bộ nhớ. Công việc được xem là hoàn tất khi giải thích được tình trạng chậm dần trên Windows và xử lý tình trạng đó bằng coverage hồi quy.