containerd / containerd/nerdctl

nerdctl ps slows down and errors with 350+ containers

Open
#325 5 comments 0 reactions 0 assignees View on GitHub
enhancement performance
Dominant language
Go
Stars
10.4k
Forks
826
Avg merge
1d 23h
Merged PRs (30d)
44

Description

I'm running a load test on my kubernetes cluster using [ClusterLoader2](https://github.com/kubernetes/perf-tests/tree/master/clusterloader2), which just runs a bunch of `pause` containers on each node. When I have around 350 containers, the performance of `nerdctl ps` is affected compared to `ctr`:

```
$ time sudo /usr/local/bin/nerdctl --debug-full -n k8s.io ps
FATA[0020] container "4690dc5561d03a5c89453546a3c5d7c0a7ce7c3938cac1560cf358a2c6c040e9" in namespace "k8s.io": not found

real 0m20.374s
user 0m0.284s
sys 0m0.117s
$ time sudo /usr/local/bin/nerdctl --debug-full -n k8s.io container ls
FATA[0034] container "4a08c9bdba3dc064ad82fbd583f992d409dd8ee3346bfd413ea010cae1f43030" in namespace "k8s.io": not found

real 0m34.376s
user 0m0.284s
sys 0m0.132s
$ time sudo ctr -n k8s.io c ls | wc -l
354

real 0m0.126s
user 0m0.081s
sys 0m0.091s
```

Also notice that the command fails due to a container being removed while the command was running (which becomes more likely the longer the command takes).

I think the race condition is caused by `ps.go` calling `c.Spec` on each container after fetching the list of containers, meaning that if a container is removed before we can inspect it, the command will error. Could be fixed by skipping the removed container rather than erroring if the error is "not found": https://github.com/containerd/nerdctl/blob/cee3b6a4840db6b5dd4019ef343af7bf4ba5c940/cmd/nerdctl/ps.go#L118-L122

Not sure what to do about the performance issue though if we have to make `O(n)` requests to `Spec` each container. Maybe we could do some of those requests in parallel?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.