microsoft / microsoft/Windows-Containers

containerd volume ownership results in access denied by HostProcessContainer

Open
#621 16 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

question triage
Dominant language
PowerShell
Stars
551
Forks
76
PR merge metrics
No merged PRs in 30d

Description

I'm trying to develop a csi driver which loads an oci image with a writable layer for use in CI builds. To accomplish this I use ctr binary to do a pull operation and then an image mount operation.

During initial prototyping this worked fine where I ran (as Administrator) containerd and the ctr commands in foreground terminals. However, when trying to run the csi driver as a HostProcessContainer the ctr image mount ... command fails with an access denied message on a volume.

I really have no idea what windows is doing behind the scenes for the snapshotter etc but I'm a bit perplexed at what would cause the hpc to not have permissions to the volume seemingly being created behind the scenes over the containerd pipe.

This is an on-prem setup using rke2 so containerd is being launched behind the scenes by rke2 service which runs as service runs under LocalSystem.

# note, as kubelet continues to retry the volumeID in the log changes every time
2025-11-04T15:01:50.767444500-05:00 csi-driver {"host":"na01wkubw01d11","level":"error","message":"handler error - driver: EphemeralInlineContainerDOciDriver method: NodePublishVolume error: {\"code\":1,\"stdout\":\"sha256:4616cf4228a0bf956c4348505b73c51f87aee27e707cf534ee08a090c12def7c\\n\",\"stderr\":\"ctr: failed to set volume mount path for layer C:\\\\var\\\\lib\\\\rancher\\\\rke2\\\\agent\\\\containerd\\\\io.containerd.snapshotter.v1.windows\\\\snapshots\\\\1396: failed to bind target \\\"\\\\\\\\\\\\\\\\?\\\\\\\\Volume{334f42c6-91e6-4b41-8420-8877bd1abed4}\\\\\\\\\\\" to root \\\"c:\\\\\\\\var\\\\\\\\lib\\\\\\\\kubelet\\\\\\\\pods\\\\\\\\f028b309d27e-421e-b951-d7efb6bc6562\\\\\\\\volumes\\\\\\\\kubernetes.io~csi\\\\\\\\oci\\\\\\\\mount\\\": Access is denied.\\n\",\"timeout\":false}","service":"democratic-csi","timestamp":"2025-11-04T20:01:50.781Z"}
  securityContext:
    windowsOptions:
      hostProcess: true
      runAsUserName: NT AUTHORITY\SYSTEM

Note, none of this would really be required if the new oci image volume type in k8s supported mounting as rw and it was somehow possible to tell/hint the snapshotter how large to make the scratch layer in windows (I need more space than the seeming default of 20GB to do the builds).

OR

If it was possible to hint via a pod label or whatever how large the scratch layer should be for the running Pod.

OR

NTFS/ReFS had some sort of CoW features like zfs et al where I could just create a rw snapshot of a whole folder on the host.

OR

Any other ideas? Everywhere I turn I keep running into walls :(

I want to use k8s containers as CI build agents (currently using with both bamboo and gh runners). I have some fairly large git repos that I don't want to:

  • pull the whole repo for every build (network traffic and build delay)
  • use N * X storage on the host where N = number of concurrent builds and X = repo size

So I need to cache the repo once (oci seems great for this, as over time I can update the image with more recent git data to lower the git delta required to pull each build) and have a way to create a rw scratch area for every build where I don't copy the full content for every build.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with src/driver/ephemeral-inline-containerd-oci/index.js#L331-L382 and the reported ctr image mount failure under a HostProcessContainer. Reproduce the access-denied volume bind in the stated RKE2 and LocalSystem setup, then determine whether the issue has a confirmed permission cause or a documented workaround for writable OCI volumes on Windows.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
infrastructure, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.