containerd / containerd/nri

Synchronize() can miss pod sandboxes that are in the process of being created, leading to missing PodSandbox events

Open
#63 4 comments 0 reactions 1 assignee Claimed by @klihub View on GitHub
v1.0 required
Dominant language
Go
Stars
406
Forks
102
Avg merge
1d 10h
Merged PRs (30d)
8

Description

We have a plugin that monitors for `RunPodSandbox` events. We observed that if a `RunPodSandbox` requests is in flight while the NRI plugin starts up and registers, then the pod sandbox event will be missed and not delivered in `Synchronize` or `RunPodSandbox`.

Here's the timeline:

1. Kubelet issues a `RunPodSandbox` creation event
2. Containerd starts to process the `RunPodSandbox`, and creates a pod sandbox in [sandboxstore.StateUnknown](https://github.com/containerd/containerd/blob/ff6324c9532b60137729a0e75fc589a2b20242b7/pkg/cri/server/sandbox_run.go#L115-L129)
3. Containerd doesn't [send a `RunPodSandbox` NRI event](https://github.com/containerd/containerd/blob/ff6324c9532b60137729a0e75fc589a2b20242b7/pkg/cri/server/sandbox_run.go#L371-L374) (because no NRI plugin is registered just yet)
4. NRI Plugin Starts up & Registers
5. containerd registers the plugin and [synchronizes](https://github.com/containerd/containerd/blob/ff6324c9532b60137729a0e75fc589a2b20242b7/pkg/nri/nri.go#L442-L445) it's state. As part of doing so, it list all the pod sandboxes, but [note it filters out sandboxes in `sandboxstore.StateUnknown`](https://github.com/containerd/containerd/blob/ff6324c9532b60137729a0e75fc589a2b20242b7/pkg/cri/nri/nri_api_linux.go#L378)
6. The NRI plugin recevies the synchronized list of PodSandboxes, but it misses the pod in (1) because the sandbox was in Unknown state
7. The `RunPodSandbox` completes
8. The `RunPodSandbox` event was missed from both `Synchronize` call and `RunPodSandbox` NRI events!

Expected behavior:

I would expect that for every pod sandbox event, it will be delivered in either `Synchronize` or `RunPodSandbox`. Maybe one approach to consider is for `Synchronize` to return pod sandboxes creations that are in flight (i.e. don't exclude Unknown state pod sandboxes).

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.