deepfence / deepfence/ThreatMapper

Package-scanner fails to mount overlayfs on kernel 5.15 with exit status 32

Open
#2,432 0 comments 0 reactions 0 assignees View on GitHub
bug needs-triage
Dominant language
TypeScript
Stars
5.3k
Forks
631
PR merge metrics
No merged PRs in 30d

Description

**Describe the bug**
The package-scanner component (part of the Deepfence agent) fails to generate SBOM for running containers on Kubernetes nodes with Linux kernel >5.11. The agent detects containerd successfully and attempts to mount the container's filesystem using overlay mount, but the mount command fails with exit status 32.
The agent tries two mounting approaches:
Direct paths from the host (/var/lib/containerd/...)
Through the mounted host filesystem (/fenced/mnt/host/var/lib/containerd/...)
Both attempts fail with the same error.

**To Reproduce**
Steps to reproduce the behavior:

1. Deploy Deepfence agent on a Kubernetes cluster with nodes running Linux kernel 5.15 (e.g., Ubuntu 22.04 LTS).
2. Ensure containerd is the container runtime.
3. Configure the agent with securityContext.privileged: true for both DaemonSet and cluster-agent (if applicable).
4. Trigger a vulnerability scan on a running container (either automatically or manually via API).
5. Check the logs of the package-scanner (or the agent pod where scanning is executed). You will see mount errors with exit status 32.

**Expected behavior**
The mount operation should succeed, allowing package-scanner to mount the container filesystem, generate the SBOM, and return results to the console.

**Screenshots**
`2026-07-23T07:48:13Z INF grpc.go:129 > SBOMRequest request={"container_id":"eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c","host_name":"tm-console-k8s-infra-worker-a02","kubernetes_cluster_name":"tm-console","node_type":"container","scan_id":"eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c-1784792889","scan_type":"base","source":"eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c"}
2026-07-23T07:48:13Z INF grpc.go:211 > Adding to map scan_id=eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c-1784792889
2026/07/23 07:48:13 [DEBUG] POST https://deepfence-console-router.cybersec.svc.cluster.local:443/deepfence/auth/token
2026-07-23T07:48:13Z INF syft.go:42 > ContainerScan container_scan={}
2026-07-23T07:48:13Z INF autodetect.go:194 > connected successfully to endpoint: unix:///run/containerd/containerd.sock
2026-07-23T07:48:13Z INF autodetect.go:194 > connected successfully to endpoint: unix:///run/containerd/containerd.sock

2026-07-23T07:48:21Z INF autodetect.go:231 > container runtime detected: containerd

2026-07-23T07:48:21Z INF containerd.go:310 > mount command: /usr/bin/bash -c mount -t overlay overlay /tmp/syft-2000182779eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c -o index=off,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1443/work,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1443/fs,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/303/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/302/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/301/fs

2026-07-23T07:48:21Z WRN containerd.go:314 > error while mounting image on temp target dir mount -t overlay overlay /tmp/syft-2000182779eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c -o index=off,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1443/work,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1443/fs,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/303/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/302/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/301/fs
err: exit status 32

2026-07-23T07:48:21Z INF containerd.go:315 > Reattempting mount from /fenced/mnt/host

2026-07-23T07:48:21Z INF containerd.go:335 > mount command: /usr/bin/bash -c mount -t overlay overlay /tmp/syft-2000182779eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c -o index=off,lowerdir=/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1443/work:/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1443/fs:/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/303/fs:/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/302/fs:/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/301/fs

2026-07-23T07:48:21Z ERR containerd.go:338 > error while mounting image on temp target dir 2nd attempt mount -t overlay overlay /tmp/syft-2000182779eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c -o index=off,lowerdir=/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1443/work:/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/1443/fs:/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/303/fs:/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/302/fs:/fenced/mnt/host/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/301/fs
err: exit status 32

2026-07-23T07:48:21Z ERR syft.go:69 > failed to extract filesystem error="exit status 32"
2026-07-23T07:48:21Z ERR syft.go:181 > failed to export filesystem error="exit status 32"
2026-07-23T07:48:21Z ERR grpc.go:234 > error in GenerateSBOM error="exit status 32"
2026-07-23T07:48:21Z INF grpc.go:221 > Removing from map scan_id=eca6181a71880c835a9e747685ba51259b4a5d2d299a3531750d2a4aab03cd3c-1784792889
2026/07/23 07:48:21 [DEBUG] POST https://deepfence-console-router.cybersec.svc.cluster.local:443/deepfence/ingest/vulnerabilities-scan-logs`

**Components/Services affected**

- [ ] UI/Frontend
- [ ] API/Backend
- [x] Agent
- [ ] Deployment/YAMLs
- [ ] CI/CD Integration
- [ ] Other (specify)

**Additional context**
The overlayfs driver in recent kernels enforces exclusive use of upperdir. If a container is running, its upperdir (the writable layer) is already in use by the container runtime. The mount call from within the agent tries to use the same upperdir (or workdir) as part of the snapshot, which triggers the protection and returns EINVAL (exit code 32).

Contributor guide

Open the contributing guide

Research direction

Start in the package-scanner paths named in the logs, especially containerd.go and syft.go, and trace the overlay mount attempts and filesystem export failure. Reproduce a scan on a Kubernetes node running Linux 5.15 with containerd, then inspect how the existing direct and /fenced/mnt/host paths behave. Done means the running container filesystem mounts successfully, an SBOM is generated, and the exit-status-32 failure is covered by a test or reproducible validation.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes, linux
Domain
infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.