containerd / containerd/containerd

metadata: daemon-wide stalls from unbounded proxy plugin calls (content GC, calls inside bolt transactions)

Open
#14,010 0 comments 0 reactions 0 assignees View on GitHub
kind/bug
Dominant language
Go
Stars
21.3k
Forks
4.1k
Avg merge
2d 4h
Merged PRs (30d)
92

Description

### Description

Several daemon-side calls into snapshotter / content-store plugins run while holding a shared lock, and on the GC paths they also drop the caller's cancellation. When one of these calls reaches a proxy plugin that stalls, the stuck RPC keeps the lock held and unrelated work across the daemon queues behind it. This is the failure mode #13798 hit in production. Its fix (#13799) covers only snapshot GC `Remove`, but the same problem exists in several other places, and how serious each one is depends on whether cancellation can reach the call.

GC paths cannot be cancelled, so they stall forever. Content GC (`contentStore.garbageCollect`) holds the content store's exclusive lock and calls the backend `Walk`/`Delete` with `context.WithoutCancel` and no deadline. A wedged content proxy then blocks all content operations (pull ingest and so on) until the plugin recovers on its own. This is the content-side twin of #13798, #13799, and it is unbounded today. Snapshot GC `Walk` is identical; #13799's author flagged it as a follow-up.

Calls inside the bolt write transaction recover on their own. `Snapshotter.Update`/`Commit` and content `Writer`/`Abort` call the backend inside a bolt write transaction and pass the request ctx through. bolt allows a single writer, so while the call is in flight every metadata write in the daemon waits behind it. The stall clears once the triggering caller's deadline fires or it disconnects.

Only calls where a hang holds a shared lock are listed below. "Bounded by" is what ends the stall:

| Call | Location (main, `6c12665e1`) | Lock held | Bounded by |
| --- | --- | --- | --- |
| content GC `Walk`/`Delete` | `content.go:916,918` | `cs.l.Lock` | **nothing** (`WithoutCancel`, `db.go:556`) — permanent |
| snapshot GC `Walk` | `snapshot.go:953` | `s.l.Lock` | **nothing** (`WithoutCancel`) — follow-up per #13799 |
| snapshot GC `Remove` | `snapshot.go:992` | `s.l.Lock` | fixed in #13799 |
| `Snapshotter.Update` | `snapshot.go:228` | bolt write tx | caller's ctx deadline |
| `Snapshotter.Commit` | `snapshot.go:642` | bolt write tx | caller's ctx deadline |
| content `Writer` (ingest) | `content.go:475` | bolt write tx | caller's ctx deadline |
| content `Abort` | `content.go:365` | bolt write tx | caller's ctx deadline |

`Snapshotter.Update`/`Commit` already carry `NOTE:` comments asking the backend to be fast "to prevent metadata store locking". The risk is known, but nothing enforces it.

### Steps to reproduce the issue

Full repro code and the goroutine dump are in the attachment below.

**1. Content GC `Delete`.** This is the permanent case. A `go test` against `main`, no root required: wrap the content store so `Delete` hangs, write one unreferenced blob, trigger GC, then start an unrelated pull ingest. GC holds `cs.l` inside the hung `Delete` (`content.go:918`), and the ingest blocks on `cs.l.RLock` (`content.go:390`). Because the ctx is `WithoutCancel`, no client deadline ends it.

**2. `Snapshotter.Update`.** This is the self-healing case. Run a real containerd with a stub proxy snapshotter that hangs in `Update`, registered as `proxy_plugins.hanging`:

```
ctr snapshot --snapshotter hanging prepare victim ""
ctr snapshot --snapshotter hanging label victim foo=bar # hangs; run with NO client timeout
ctr snapshot --snapshotter overlayfs prepare bystander "" # BLOCKS: unrelated snapshotter
```

The `overlayfs` prepare does not return while the `Update` is in flight. The profile shows the hung `Update` holding the bolt write tx and the prepare blocked in `beginRWTx`. The `label` client must run without a timeout: gRPC is context-aware, so a client disconnect unwinds the `Update` and releases the writer. That is what bounds this case to the caller, whereas case 1 has no such bound.

### Describe the results you received and expected

Received: a wedged content proxy blocks all content operations for as long as the plugin stays stuck (case 1), and a wedged snapshotter proxy blocks all metadata writes for as long as the caller waits (case 2).

Expected: a single unresponsive proxy plugin should degrade only that plugin. The GC paths in particular must not hold a store lock forever with cancellation stripped.

For a fix, I would start with content GC, bounding it the same way #13799 bounds snapshot GC (`io.containerd.timeout.gc.content.*`). The tx-internal calls already self-heal and may not need any change. A blanket interceptor timeout is not appropriate, since `diff.Apply` and streaming calls are legitimately long-running.

### What version of containerd are you using?

containerd github.com/containerd/containerd/v2 v2.3.0-266-g1ab181cb6 1ab181cb645ad156606a68d3fca83dd59ebf7fd8

### Any other relevant information

_No response_

### Show configuration if it is related to CRI plugin.

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with content.go:916 and 918, then compare snapshot.go:953 with the timeout approach used for snapshot GC Remove in #13799. Run the described go test reproduction for a hanging content Delete and inspect the listed lock paths. Done means a stalled proxy no longer holds the content or snapshot store lock indefinitely, while unrelated operations can proceed.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.