Linkerd Namespace stuck in Deletion
- Dominant language
- JavaScript
- Stars
- 60
- Forks
- 221
- Avg merge
- 4h 32m
- Merged PRs (30d)
- 1
Description
Hi Folks,
k8 v1.17.3 (experienced since 1.14 I think)
This is more of a call to add something to the docs (because it turns out to be a somewhat odd issue that you might not easily find) than it is a problem with linkerd.
So when deleting linkerd2 it automagically attempts to delete the namespace. The namespace will be stuck in the terminating state and never deleted. Here's the replication steps:
1. Install Linkerd
2. Install prometheus adapter (this is for HPA metrics)
3. Uninstall linkerd (attempts to delete the namespace)
4. Run `kubectl get ns linkerd -o yaml` to see that it's stuck in the terminating phase with "kubernetes" as a finalizer
Conventional (a.k.a. [blog posting](https://medium.com/@clouddev.guru/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e) or stack overflow) wisdom dictates that you should do a little hack to remove the kubernetes finalizer allowing the namespace to be deleted. However I think this is not the correct way to resolve this issue. Here's why:
If you go ahead and run `kubectl api-resources` you might see a cheeky little error down the bottom:

Which promptly produces in 99% of k8 admins a WTF response. Upon further investigation you find that if you run `kubectl get apiservice` we see something quite interesting:

The custom metrics services are failing because the prometheus adapter no longer exists. You wouldn't think this because in all likelihood you've run some sort of k8 foo to check if there are any resources left in the linkerd namespace and found it empty.
So once you go ahead and run `kubectl delete apiservice v1beta1.custom.metrics.k8s.io` then you'd assume things were all good. They're not, because you have to be patient for about 5 minutes or so and then the namespace will finally be deleted (and perhaps longer if there's some sort of exponential timeout on the finaliser, smarter k8 folks than I can answer that question). There's also a chance that you don't wanna delete the apiservice if there are other services that declare it, but it's not something I've come across yet so I can't speak to it.
So what's the solution needed here?
I'm not entirely sure, but I know that lots of people use linkerd with the prometheus-operator as I assume that's a pretty common use pattern. I think someone with better k8 experience needs to talk about what happens when an apiservice (v1beta1.custom.metrics.k8s.io) is associated with multiple services like `linkerd/prometheus-adapter`. Then adding some sort of discussion to "uninstalling linkerd" should be added to make sure any custom piece installed into the linkerd namespace is uninstalled first.
Contributor guide
Assessment
This issue has not been assessed yet.