microsoft / microsoft/Windows-Containers

CimFS snapshotter causes BSOD on Windows Server 2022 (build 20348.4773) under concurrent container workload

Open
#633 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
PowerShell
Stars
551
Forks
76
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

Using the CimFS snapshotter (cimfs.sys) with containerd 2.2.1 on Windows Server 2022 causes repeated BSODs when pulling and starting multiple container images concurrently. Two distinct bugchecks occurred within 13 minutes:

  1. 0x00000044 (MULTIPLE_IRP_COMPLETE_REQUESTS)
  2. 0x000001ca (SYNTHETIC_WATCHDOG_TIMEOUT)

Also filed as https://github.com/microsoft/hcsshim/issues/2625 since the CimFS snapshotter/differ implementation lives in hcsshim.

To Reproduce

  1. Install containerd v2.2.1 on Windows Server 2022 (build 20348.4773).
  2. Configure containerd to use the cimfs snapshotter:
    version = 3
    
    [plugins.'io.containerd.cri.v1.images']
      snapshotter = 'cimfs'
    
    [plugins.'io.containerd.transfer.v1.local']
      [[plugins.'io.containerd.transfer.v1.local'.unpack_config]]
        platform = 'windows/amd64'
        snapshotter = 'cimfs'
        differ = 'cimfs'
    
  3. Start containerd and kubelet.
  4. Pull ~15-20 Windows container images concurrently (Kubernetes daemonsets + workloads).
  5. System BSODs within minutes.

Expected behavior

The CimFS kernel driver should handle concurrent image layer mounts without crashing.

Configuration:

  • Edition: Windows Server 2022 Datacenter (21H2)
  • OS Build: 20348.4773 (KB5075906, February 2026 cumulative update)
  • cimfs.sys driver version: 10.0.20348.3804
  • Container engine: containerd v2.2.1
  • Orchestrator: Kubernetes v1.34.2
  • Hardware: Bare metal, AMD Ryzen 9 7950X, 64GB RAM
  • Base images: mcr.microsoft.com/windows/servercore:ltsc2022 and mcr.microsoft.com/windows/nanoserver:ltsc2022

Additional context

Bugcheck details:

Bugcheck 1 (12:37 PST):
  Code: 0x00000044 (MULTIPLE_IRP_COMPLETE_REQUESTS)
  Parameters: (0xffffb1075846f4c0, 0x000000000000125a, 0x0000000000000000, 0x0000000000000000)

Bugcheck 2 (12:50 PST):
  Code: 0x000001ca (SYNTHETIC_WATCHDOG_TIMEOUT)
  Parameters: (0x0000000019628f83, 0x00000001bf799269, 0x00000001bf7ca01d, 0x0000000000000008)

The system was stable for over 3 years with the default windows (WCIFS) snapshotter. BSODs started immediately after switching to CimFS and stopped after reverting. The crashes appear to be in the cimfs.sys kernel driver under concurrent I/O load.

Minidumps are available if needed.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the CimFS snapshotter configuration and concurrent workload reproduction described in the issue, then inspect the available minidumps. Check the related hcsshim issue, since the CimFS snapshotter and differ implementation live there. Done means identifying and addressing the CimFS or integration cause of the crashes, with concurrent pulls no longer producing BSODs.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
devops, infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.