google / google/gvisor

Debugging with dlv inside gvisor with systrap faults

Open
#11,649 11 comments 2 reactions 0 assignees View on GitHub
platform:systrap status: help wanted type: bug
Dominant language
Go
Stars
19.3k
Forks
2k
Avg merge
3d 5h
Merged PRs (30d)
264

Description

### Description

Hey gvisor folks !

I have a use case where I need to debug a go binary with dlv, running w/ gvisor.
When I set a breakpoint, reach it, and then try to step - dlv is either stuck or faults.

This only happens when running with systrap, using --platform=ptrace, it works as expected;
as the ptrace platform usage is discouraged, I would really appreciate your help w/ making it work under systrap.

It happens with the latest runsc, go and dlv releases.

### Steps to reproduce

```sh
mkdir /tmp/gvisor-dlv
cd /tmp/gvisor-dlv

# get runsc
ARCH=$(uname -m)
wget -q https://storage.googleapis.com/gvisor/releases/release/latest/$ARCH/runsc
chmod +x runsc
export PATH=$PATH:/tmp/gvisor-dlv

# get go binary
wget -qO- https://go.dev/dl/go1.24.2.linux-$(echo $ARCH | sed -E 's/x86_/amd/;s/aarch/arm/').tar.gz | tar xzf -
export PATH=$PATH:/tmp/gvisor-dlv/go/bin

# create oci bundle
mkdir bundle
mkdir --mode=0755 bundle/rootfs

# get dlv binary
GOBIN=/tmp/gvisor-dlv/bundle/rootfs go install github.com/go-delve/delve/cmd/dlv@latest

# compile go binary
go mod init main
cat < main.go
package main

func main() {
for i := range 1000 {
print(i)
}
}
EOF
go build -gcflags 'all=-N -l' -o bundle/rootfs/main .

# create mount dir for uds
mkdir mnt
chmod 777 mnt

# create oci config
cat < bundle/config.json
{
"ociVersion": "1.0.0",
"process": {
"args": [
"/dlv",
"--listen=unix:/mnt/dlv.sock",
"--headless",
"--api-version=2",
"--accept-multiclient",
"exec",
"/main"
]
},
"root": {
"path": "rootfs"
},
"mounts": [
{
"destination": "/mnt",
"type": "bind",
"source": "/tmp/gvisor-dlv/mnt"
}
]
}
EOF

# run gvisor
sudo -E env PATH=$PATH runsc --host-uds=all run -bundle /tmp/gvisor-dlv/bundle test
# > API server listening at: /mnt/dlv.sock

# connect to dlv in another session
cd /tmp/gvisor-dlv
sudo chmod 666 mnt/dlv.sock
bundle/rootfs/dlv connect unix:mnt/dlv.sock
# > (dlv)

# set breakpoint
(dlv) b main.main
# > Breakpoint 1 set at 0x470b0a for main.main() ./main.go:3

# reach breakpoint
(dlv) c
# > [Breakpoint 1] main.main() ./main.go:3 (hits goroutine(1):1 total:1) (PC: 0x470b0a)

# next
(dlv) n

# dlv should now hang or fault, it might take a few runs to hit the fault flow;
# on rare occasions it works properly, if so rerunning will usually trigger the hang / fault
# > unexpected fault address 0xc0000547d0
#> [signal SIGSEGV: segmentation violation code=0x2 addr=0xc0000547d0 pc=0xc0000547d0]

# > [runtime-fatal-throw] runtime.throw() ./go/src/runtime/panic.go:1092 (hits goroutine(1):1 total:1) (PC: # 0x468084)

```

### runsc version

```shell
runsc version release-20250414.0
spec: 1.1.0-rc.1
```

### uname

```shell
Linux ip-10-0-1-16 6.8.0-1024-aws #26-Ubuntu SMP Tue Feb 18 17:22:37 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
```

### runsc debug logs

[runsc-debug-log.tar.gz](https://github.com/user-attachments/files/19801478/runsc-debug-log.tar.gz)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.