kubernetes-sigs / kubernetes-sigs/cluster-api
RuntimeSDK: Potential deadlock after ExtensionConfig deletion
- Dominant language
- Go
- Stars
- 4.3k
- Forks
- 1.6k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 113
Description
Disclaimer: We are not sure if this deadlock is actually possible. It could be possible, we can either wait until it was actually observed or investigate controller-runtime if it is possible / can be intentionally reproduced.
Some context:
* The `ExtensionConfig` controller discovers extensions by sending Discovery requests to the service/url configured in an ExtensionConfig
* This discovery information is then registered in a local registry
* When we e.g. want to call all extensions for a `BeforeClusterCreate` hook we query the registry and then call all registered extensions
The deadlock can happen in the following situation:
* User deletes the Runtime Extension including the ExtensionConfig and the corresponding Deployment / Service
* The `ExtensionConfig` controller does not get a Reconcile call for the delete event
* This is the part that we are not sure about!
* When we now want to call extensions for a hook this could include an already removed extension. The call will thus always fail.
I think this case is possible if it's possible that:
* A watch fails / or misses an event
* Subsequent list/watches don't retrieve an already deleted object
* A `Delete` event is never send to the `ExtensionConfig` controller
When we want to address this issue, let's definitely first figure out if it can happen and how we can reproduce it. This will require some investigation in controller-runtime
Some ideas to address the issue then:
* We have to make sure that we are reflecting an ExtensionConfig deletion in all cases in the registry (even if we miss a deletion event)
* Solutions could be:
* implement a periodic check to verify all registry entries still have a corresponding ExtensionConfig
* add a finalizer (we have to think about what happens with multiple controller replicas)
**Environment:**
- Cluster-api version: main
/kind bug
[One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels]
Contributor guide
Assessment
This issue has not been assessed yet.