microsoft / microsoft/Windows-Containers
CimFS snapshotter causes BSOD on Windows Server 2022 (build 20348.4773) under concurrent container workload
Nobody has claimed this yet.
- Dominant language
- PowerShell
- Stars
- 551
- Forks
- 76
- PR merge metrics
- No merged PRs in 30d
Description
Describe the bug
Using the CimFS snapshotter (cimfs.sys) with containerd 2.2.1 on Windows Server 2022 causes repeated BSODs when pulling and starting multiple container images concurrently. Two distinct bugchecks occurred within 13 minutes:
0x00000044(MULTIPLE_IRP_COMPLETE_REQUESTS)0x000001ca(SYNTHETIC_WATCHDOG_TIMEOUT)
Also filed as https://github.com/microsoft/hcsshim/issues/2625 since the CimFS snapshotter/differ implementation lives in hcsshim.
To Reproduce
- Install containerd v2.2.1 on Windows Server 2022 (build 20348.4773).
- Configure containerd to use the
cimfssnapshotter:version = 3 [plugins.'io.containerd.cri.v1.images'] snapshotter = 'cimfs' [plugins.'io.containerd.transfer.v1.local'] [[plugins.'io.containerd.transfer.v1.local'.unpack_config]] platform = 'windows/amd64' snapshotter = 'cimfs' differ = 'cimfs' - Start containerd and kubelet.
- Pull ~15-20 Windows container images concurrently (Kubernetes daemonsets + workloads).
- System BSODs within minutes.
Expected behavior
The CimFS kernel driver should handle concurrent image layer mounts without crashing.
Configuration:
- Edition: Windows Server 2022 Datacenter (21H2)
- OS Build: 20348.4773 (KB5075906, February 2026 cumulative update)
cimfs.sysdriver version: 10.0.20348.3804- Container engine: containerd v2.2.1
- Orchestrator: Kubernetes v1.34.2
- Hardware: Bare metal, AMD Ryzen 9 7950X, 64GB RAM
- Base images:
mcr.microsoft.com/windows/servercore:ltsc2022andmcr.microsoft.com/windows/nanoserver:ltsc2022
Additional context
Bugcheck details:
Bugcheck 1 (12:37 PST):
Code: 0x00000044 (MULTIPLE_IRP_COMPLETE_REQUESTS)
Parameters: (0xffffb1075846f4c0, 0x000000000000125a, 0x0000000000000000, 0x0000000000000000)
Bugcheck 2 (12:50 PST):
Code: 0x000001ca (SYNTHETIC_WATCHDOG_TIMEOUT)
Parameters: (0x0000000019628f83, 0x00000001bf799269, 0x00000001bf7ca01d, 0x0000000000000008)
The system was stable for over 3 years with the default windows (WCIFS) snapshotter. BSODs started immediately after switching to CimFS and stopped after reverting. The crashes appear to be in the cimfs.sys kernel driver under concurrent I/O load.
Minidumps are available if needed.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the CimFS snapshotter configuration and concurrent workload reproduction described in the issue, then inspect the available minidumps. Check the related hcsshim issue, since the CimFS snapshotter and differ implementation live there. Done means identifying and addressing the CimFS or integration cause of the crashes, with concurrent pulls no longer producing BSODs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- devops, infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100