Synchronize() can miss pod sandboxes that are in the process of being created, leading to missing PodSandbox events
- Dominant language
- Go
- Stars
- 406
- Forks
- 102
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 8
Description
We have a plugin that monitors for `RunPodSandbox` events. We observed that if a `RunPodSandbox` requests is in flight while the NRI plugin starts up and registers, then the pod sandbox event will be missed and not delivered in `Synchronize` or `RunPodSandbox`.
Here's the timeline:
1. Kubelet issues a `RunPodSandbox` creation event
2. Containerd starts to process the `RunPodSandbox`, and creates a pod sandbox in [sandboxstore.StateUnknown](https://github.com/containerd/containerd/blob/ff6324c9532b60137729a0e75fc589a2b20242b7/pkg/cri/server/sandbox_run.go#L115-L129)
3. Containerd doesn't [send a `RunPodSandbox` NRI event](https://github.com/containerd/containerd/blob/ff6324c9532b60137729a0e75fc589a2b20242b7/pkg/cri/server/sandbox_run.go#L371-L374) (because no NRI plugin is registered just yet)
4. NRI Plugin Starts up & Registers
5. containerd registers the plugin and [synchronizes](https://github.com/containerd/containerd/blob/ff6324c9532b60137729a0e75fc589a2b20242b7/pkg/nri/nri.go#L442-L445) it's state. As part of doing so, it list all the pod sandboxes, but [note it filters out sandboxes in `sandboxstore.StateUnknown`](https://github.com/containerd/containerd/blob/ff6324c9532b60137729a0e75fc589a2b20242b7/pkg/cri/nri/nri_api_linux.go#L378)
6. The NRI plugin recevies the synchronized list of PodSandboxes, but it misses the pod in (1) because the sandbox was in Unknown state
7. The `RunPodSandbox` completes
8. The `RunPodSandbox` event was missed from both `Synchronize` call and `RunPodSandbox` NRI events!
Expected behavior:
I would expect that for every pod sandbox event, it will be delivered in either `Synchronize` or `RunPodSandbox`. Maybe one approach to consider is for `Synchronize` to return pod sandboxes creations that are in flight (i.e. don't exclude Unknown state pod sandboxes).
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.