containerd / containerd/nerdctl

Successive calls to `client.Containers(ctx)` then `c.Labels` may fail with `ErrNotFound` (possibly other methods as well)

Open
#3,167 0 comments 0 reactions 0 assignees View on GitHub
kind/unconfirmed-bug-claim
Dominant language
Go
Stars
10.4k
Forks
826
Avg merge
1d 23h
Merged PRs (30d)
44

Description

### Description

Unfortunately, it seems to be widespread across the codebase: we do make the assumption that whatever containers we get from `client.Containers` is still available down the road for other operations (like reading the Labels of the container).

This is of course not true. By the time we read the Labels, the container may no longer exist.

For a concrete example, see: https://github.com/containerd/nerdctl/blob/main/pkg/cmd/volume/rm.go#L34-L76

The problem is that this will make unrelated operations fail (that should not fail), when we hard error on that.
In the specific case I am pointing out above, this will make `volume rm foo` fail, even (especially) if the container in question has no relation to that volume whatsoever.

I am going to fix this particular instance, because it trips test parallelization I am working on, but there is obviously much more in there.

I do not have a good generic solution - introducing a global locking mechanism for nerdctl is likely a bad idea - it will significantly slow down concurrency performance, and will not even address some cases.

Also unfortunately, this is very tricky to reproduce and there is no simple way to trip these conditions unless we massively parallelize all our tests (which I want to do) and tackle them one by one...

The only solution I can think of is a careful review of the codebase to spot all these places where we are making assumptions about the availability of containers.

### Steps to reproduce the issue

na

### Describe the results you received and expected

na

### What version of nerdctl are you using?

main

### Are you using a variant of nerdctl? (e.g., Rancher Desktop)

None

### Host information

_No response_

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.