canonical / canonical/workshop

LXD socket tunnel (`tunnel` interface) broken since 0.9.4: `lxc` inside a workshop fails with "connection reset by peer"

Open
#971 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
108
Forks
17
Avg merge
1d 7h
Merged PRs (30d)
40

Description

**Component:** workshop snap, `lxd-tunnel` SDK / `tunnel` interface
**Severity:** High — `lxc` (and any LXD client) inside a workshop is completely unusable
**Affected versions:** workshop **0.9.4** and later (regression)
**Introduced by:** commit `dbbca6369325e3d6cb0cee2db090ba7009909f1f` — "Drop privileges for tunnels from workshop to host" (2026-07-09)

## Environment

- Host: Ubuntu 26.04 LTS, kernel `7.0.0-28-generic` (x86_64)
- workshop: snap **0.9.4-ebb8bb7** (rev 386, `latest/stable`, `classic`)
- LXD: snap **6.9** (host daemon at `/var/snap/lxd/common/lxd/unix.socket`)
- Host user: uid 10000, **primary** gid 10000; `lxd` group gid 114, membership **supplementary only** (`getent group lxd` → `lxd:x:114:myuser`)

## Summary

Since workshop 0.9.4, a workshop that tunnels the host LXD socket into the container exposes the socket, but **every connection is reset**. Inside the container:

```
$ lxc list
Error: LXD unix socket "/var/lib/workshop/run/host-lxd/unix.socket" not accessible: Get "http://unix.socket/1.0": read unix @->/var/lib/workshop/run/host-lxd/unix.socket: read: connection reset by peer
```

`curl` through the same tunneled socket fails identically:

```
$ curl --unix-socket /var/lib/workshop/run/host-lxd/unix.socket http://unix.socket/1.0
curl: (56) Recv failure: Connection reset by peer
```

Direct access to the real host socket works fine:

```
$ curl --unix-socket /var/snap/lxd/common/lxd/unix.socket http://unix.socket/1.0
{"api_status":"stable", ...}
```

## Steps to reproduce

### 1. Define a workshop with a `tunnel` interface to the host LXD socket

`workshop.yaml` (abridged — only the tunnel-relevant parts):

```yaml
name: dev
base: ubuntu@24.04

sdks:
- name: system
slots:
lxd:
interface: tunnel
endpoint: /var/snap/lxd/common/lxd/unix.socket
# ... other SDKs (go, node, rust, opencode, vscode-remote ...)

connections:
- plug: lxd-tunnel:lxd
slot: system:lxd
```

Where `lxd-tunnel` is a project SDK whose plug attaches the tunnel to the container:

`.workshop/lxd-tunnel/sdk.yaml`:

```yaml
name: lxd-tunnel
plugs:
lxd:
interface: tunnel
endpoint: /var/lib/workshop/run/host-lxd/unix.socket
```

### 2. Launch the workshop

```
$ workshop launch dev
```

### 3. Try to use LXD from inside the container

```
$ workshop exec dev -- lxc list
```

**Expected:** list of host LXD instances/projects.

**Actual:**

```
Error: LXD unix socket "/var/lib/workshop/run/host-lxd/unix.socket" not accessible: Get "http://unix.socket/1.0": read unix @->/var/lib/workshop/run/host-lxd/unix.socket: read: connection reset by peer
```

The failure is on the *first* request — any LXD client (Go SDK, `curl`) is affected.

## Root cause

The tunnel proxy device created for the workshop carries the host user's credentials as `security.uid`/`security.gid`:

```
$ lxc --project workshop.10000 profile show dev-6bc2a0cf-lxd-tunnel
devices:
lxd-tunnel_lxd:
bind: instance
connect: unix:/var/snap/lxd/common/lxd/unix.socket
gid: "1000"
listen: unix:/var/lib/workshop/run/host-lxd/unix.socket
security.gid: "10000"
security.uid: "10000"
type: proxy
uid: "1000"
```

LXD spawns `forkproxy` for the *connect* side of this `bind: instance` proxy. Observed on the host, that process runs as the host user with **no supplementary groups**:

```
$ ps -o user,pid,args -p
didier.+ ... /snap/lxd/current/sbin/lxd forkproxy -- ... unix:/var/lib/workshop/run/host-lxd/unix.socket ... unix:/var/snap/lxd/common/lxd/unix.socket 1000 1000 0644 10000 10000

$ grep -E "Uid|Gid|Groups" /proc//status
Uid: 10000 10000 10000 10000
Gid: 10000 10000 10000 10000
Groups:
```

The host LXD socket is owned `srw-rw---- root:lxd` (0660):

```
$ ls -l /var/snap/lxd/common/lxd/unix.socket
srw-rw---- 1 root lxd 0 Aug 3 16:52 /var/snap/lxd/common/lxd/unix.socket
```

For this user, `lxd` access is a **supplementary** group. Once `forkproxy`'s gid is the user's **primary** gid (10000), the supplementary `lxd` membership is gone, the connect to the 0660 `root:lxd` socket fails with `EACCES`, `forkproxy` aborts and closes the server-side connection, and the client — which has already written its request — sees `connection reset by peer`.

This is a direct consequence of the 0.9.4 hardening change, which adds exactly these two keys to every workshop→host tunnel device:

```go
// internal/workshop/lxd/lxd_backend.go (proxyToLxdDevice)
case workshop.WorkshopToHost:
device["bind"] = "instance"
device["security.uid"] = usr.Uid
device["security.gid"] = usr.Gid
```

```go
// internal/interfaces/lxd_device/spec.go (addProxyEntry)
device["security.uid"] = s.User.Uid
device["security.gid"] = s.User.Gid
```

`usr`/`s.User` come from `os/user.Lookup` (`internal/osutil/user.go`), i.e. the *primary* uid/gid from `/etc/passwd`.

### Why it affects only some setups

- **Broken:** host user whose **primary** gid is *not* the `lxd` group gid — i.e. anyone added to `lxd` with `adduser $USER lxd` (a supplementary group). This is the common case.
- **Works by staleness:** an unaffected machine, running the same workshop 0.9.4, has a tunnel device created *before* the upgrade that carries **no** `security.uid`/`security.gid` keys, so `forkproxy` keeps its previous (root) privileges and the tunnel works:

```
$ lxc --project workshop.1993010624 profile show dev-ffa5204a-lxd-tunnel
devices:
lxd-tunnel_lxd:
bind: instance
connect: unix:/var/snap/lxd/common/lxd/unix.socket
listen: unix:/var/lib/workshop/run/host-lxd/unix.socket
type: proxy
```

Recreating that workshop under 0.9.4 reproduces the bug.

Contributor guide

Open the contributing guide

Research direction

Start with internal/workshop/lxd/lxd_backend.go at proxyToLxdDevice and internal/interfaces/lxd_device/spec.go at addProxyEntry; inspect how tunnel proxy credentials are derived, then review internal/osutil/user.go. Reproduce with the workshop.yaml configuration and workshop launch/exec commands. Done means the tunneled host LXD socket accepts the first lxc or curl request for supplementary-group users, with regression coverage if the repository provides a suitable test location.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
56/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.