kubernetes-sigs / kubernetes-sigs/cluster-api

RuntimeSDK: Potential deadlock after ExtensionConfig deletion

Open
#6,863 12 comments 0 reactions 1 assignee Claimed by @sbueringer View on GitHub
area/runtime-sdk help wanted kind/bug priority/important-longterm triage/accepted
Dominant language
Go
Stars
4.3k
Forks
1.6k
Avg merge
1d 3h
Merged PRs (30d)
113

Description

Disclaimer: We are not sure if this deadlock is actually possible. It could be possible, we can either wait until it was actually observed or investigate controller-runtime if it is possible / can be intentionally reproduced.

Some context:
* The `ExtensionConfig` controller discovers extensions by sending Discovery requests to the service/url configured in an ExtensionConfig
* This discovery information is then registered in a local registry
* When we e.g. want to call all extensions for a `BeforeClusterCreate` hook we query the registry and then call all registered extensions

The deadlock can happen in the following situation:
* User deletes the Runtime Extension including the ExtensionConfig and the corresponding Deployment / Service
* The `ExtensionConfig` controller does not get a Reconcile call for the delete event
* This is the part that we are not sure about!
* When we now want to call extensions for a hook this could include an already removed extension. The call will thus always fail.

I think this case is possible if it's possible that:
* A watch fails / or misses an event
* Subsequent list/watches don't retrieve an already deleted object
* A `Delete` event is never send to the `ExtensionConfig` controller

When we want to address this issue, let's definitely first figure out if it can happen and how we can reproduce it. This will require some investigation in controller-runtime

Some ideas to address the issue then:
* We have to make sure that we are reflecting an ExtensionConfig deletion in all cases in the registry (even if we miss a deletion event)
* Solutions could be:
* implement a periodic check to verify all registry entries still have a corresponding ExtensionConfig
* add a finalizer (we have to think about what happens with multiple controller replicas)

**Environment:**

- Cluster-api version: main

/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.