linkerd / linkerd/website

Linkerd Namespace stuck in Deletion

Open
#712 13 comments 0 reactions 1 assignee Claimed by @cpretzer View on GitHub
Dominant language
JavaScript
Stars
60
Forks
221
Avg merge
4h 32m
Merged PRs (30d)
1

Description

Hi Folks,

k8 v1.17.3 (experienced since 1.14 I think)

This is more of a call to add something to the docs (because it turns out to be a somewhat odd issue that you might not easily find) than it is a problem with linkerd.

So when deleting linkerd2 it automagically attempts to delete the namespace. The namespace will be stuck in the terminating state and never deleted. Here's the replication steps:

1. Install Linkerd
2. Install prometheus adapter (this is for HPA metrics)
3. Uninstall linkerd (attempts to delete the namespace)
4. Run `kubectl get ns linkerd -o yaml` to see that it's stuck in the terminating phase with "kubernetes" as a finalizer

Conventional (a.k.a. [blog posting](https://medium.com/@clouddev.guru/how-to-fix-kubernetes-namespace-deleting-stuck-in-terminating-state-5ed75792647e) or stack overflow) wisdom dictates that you should do a little hack to remove the kubernetes finalizer allowing the namespace to be deleted. However I think this is not the correct way to resolve this issue. Here's why:

If you go ahead and run `kubectl api-resources` you might see a cheeky little error down the bottom:
![image](https://user-images.githubusercontent.com/674765/79291555-4f03a800-7f12-11ea-90ed-ddc4dab86fb8.png)

Which promptly produces in 99% of k8 admins a WTF response. Upon further investigation you find that if you run `kubectl get apiservice` we see something quite interesting:

![image](https://user-images.githubusercontent.com/674765/79291467-182d9200-7f12-11ea-97bd-a7d49831f764.png)

The custom metrics services are failing because the prometheus adapter no longer exists. You wouldn't think this because in all likelihood you've run some sort of k8 foo to check if there are any resources left in the linkerd namespace and found it empty.

So once you go ahead and run `kubectl delete apiservice v1beta1.custom.metrics.k8s.io` then you'd assume things were all good. They're not, because you have to be patient for about 5 minutes or so and then the namespace will finally be deleted (and perhaps longer if there's some sort of exponential timeout on the finaliser, smarter k8 folks than I can answer that question). There's also a chance that you don't wanna delete the apiservice if there are other services that declare it, but it's not something I've come across yet so I can't speak to it.

So what's the solution needed here?

I'm not entirely sure, but I know that lots of people use linkerd with the prometheus-operator as I assume that's a pretty common use pattern. I think someone with better k8 experience needs to talk about what happens when an apiservice (v1beta1.custom.metrics.k8s.io) is associated with multiple services like `linkerd/prometheus-adapter`. Then adding some sort of discussion to "uninstalling linkerd" should be added to make sure any custom piece installed into the linkerd namespace is uninstalled first.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.