microsoft / microsoft/hcsshim

BSOD (MULTIPLE_IRP_COMPLETE_REQUESTS, SYNTHETIC_WATCHDOG_TIMEOUT) when using CimFS snapshotter on Windows Server 2022

Open
#2,625 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
694
Forks
304
Avg merge
1d 19h
Merged PRs (30d)
28

Description

Describe the bug

Using the CimFS snapshotter with containerd 2.2.1 on Windows Server 2022 (build 20348.4773, KB5075906 Feb 2026 cumulative update) causes repeated BSODs under container workload. Two distinct bugchecks occurred within 13 minutes of enabling CimFS:

  1. 0x00000044 (MULTIPLE_IRP_COMPLETE_REQUESTS). a driver attempted to complete an IRP that was already completed.
  2. 0x000001ca (SYNTHETIC_WATCHDOG_TIMEOUT). system became unresponsive.

Both crashes happened while Kubernetes was pulling and starting ~15-20 container images simultaneously using the CimFS snapshotter. The system was stable before switching from the windows snapshotter to cimfs and stable again after reverting.

To Reproduce

  1. Install containerd v2.2.1 on Windows Server 2022 (build 20348.4773).
  2. Configure containerd to use the cimfs snapshotter:
    version = 3
    
    [plugins.'io.containerd.cri.v1.images']
      snapshotter = 'cimfs'
    
    [plugins.'io.containerd.transfer.v1.local']
      [[plugins.'io.containerd.transfer.v1.local'.unpack_config]]
        platform = 'windows/amd64'
        snapshotter = 'cimfs'
        differ = 'cimfs'
    
  3. Start containerd and kubelet.
  4. Pull multiple Windows container images concurrently (~15-20 images pulled by Kubernetes daemonsets and workloads).
  5. System BSODs within minutes.

Expected behavior

The CimFS snapshotter should handle concurrent image pulls and container creation without causing kernel crashes.

Configuration:

  • Edition: Windows Server 2022 Datacenter (21H2)
  • OS Build: 20348.4773 (KB5075906, February 2026 cumulative update)
  • cimfs.sys driver version: 10.0.20348.3804
  • Container engine: containerd v2.2.1
  • Container runtime shim: containerd-shim-runhcs-v1 (bundled with containerd 2.2.1)
  • Orchestrator: Kubernetes v1.34.2 (kubelet)
  • Hardware: Bare metal, AMD Ryzen 9 7950X, 64GB RAM
  • Base images: Various mcr.microsoft.com/windows/servercore:ltsc2022 and mcr.microsoft.com/windows/nanoserver:ltsc2022 based images

Additional context

Bugcheck details from Windows Event Log:

Bugcheck 1 (12:37:07 PST):
  Code: 0x00000044 (MULTIPLE_IRP_COMPLETE_REQUESTS)
  Parameters: (0xffffb1075846f4c0, 0x000000000000125a, 0x0000000000000000, 0x0000000000000000)
  Minidump: 030526-8984-01.dmp (1.4 MB)

Bugcheck 2 (12:50:18 PST):
  Code: 0x000001ca (SYNTHETIC_WATCHDOG_TIMEOUT)
  Parameters: (0x0000000019628f83, 0x00000001bf799269, 0x00000001bf7ca01d, 0x0000000000000008)
  Minidump: 030526-9125-01.dmp (630 KB)

Timeline:

  • Switched containerd snapshotter from windows to cimfs and restarted services
  • Kubernetes began pulling ~15-20 container images concurrently (daemonset pods + workloads)
  • CimFS snapshots were being created successfully (55+ active snapshots observed)
  • Several containers started successfully with CimFS layers before the crash
  • First BSOD at 12:37 (MULTIPLE_IRP_COMPLETE_REQUESTS)
  • System rebooted, containers began re-pulling images
  • Second BSOD at 12:50 (SYNTHETIC_WATCHDOG_TIMEOUT)
  • Reverted to windows snapshotter. system has been stable since

The CimFS snapshotter was loading correctly (io.containerd.snapshotter.v1 cimfs ok, io.containerd.differ.v1 cimfs ok) and creating snapshots before the crashes. The issue appears to be in the cimfs.sys kernel driver under concurrent load.

No prior bugchecks on this system since November 2022 (over 3 years of stability). Minidumps are available if needed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported containerd 2.2.1 and CimFS configuration under concurrent Windows image pulls, then review the available minidumps and hcsshim's CimFS-related entry points. Done means determining whether the failure is actionable in hcsshim or belongs to the Windows CimFS driver, with a confirmed reproduction and diagnosis documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
devops, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.