Azure / Azure/azure-cli-extensions
Az Monitor for Arc k8s extension deletion/reinstall fails
- Dominant language
- Python
- Stars
- 454
- Forks
- 1.7k
- Avg merge
- 2d 19h
- Merged PRs (30d)
- 64
Description
- If the issue is to do with Azure CLI 2.0 in-particular, create an issue here at [Azure/azure-cli](https://github.com/Azure/azure-cli/issues)
### Extension name (the extension in question)
k8s-extension
### Description of issue (in as much detail as possible)
az k8s-extension delete does not delete rsources from the cluster
The extnesion is displayed as FAILEd then in the Azure portal with errors :
```console
Error: {failed to install chart from path [] for release [azuremonitor-containers]: err [rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "omsagent" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "azuremonitor-containers": current value is "azmon-containers-release-1"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "azuremonitor-containers": current value is "default"]} occurred while doing the operation : {Installing the extension} on the config
```
```console
Error: {failed to install chart from path [] for release [azuremonitor-containers]: err [rendered manifests contain a resource that already exists. Unable to continue with install: Secret "omsagent-secret" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "azuremonitor-containers": current value is "azmon-containers-release-1"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "azuremonitor-containers": current value is "default"]} occurred while doing the operation : {Installing the extension} on the config
```
-----
```sh
az version
```
```console
{
"azure-cli": "2.22.1",
"azure-cli-core": "2.22.1",
"azure-cli-telemetry": "1.0.6",
"extensions": {
"connectedk8s": "1.1.3",
"connectedmachine": "0.3.0",
"k8s-configuration": "1.0.0",
"k8s-extension": "0.2.1",
"spring-cloud": "2.3.0"
}
}
```
I di test it on a K3S cluster deployed on an Azure VM
workaround : I had to manually clean-up the environment :
```sh
az k8s-extension delete --name azuremonitor-containers --cluster-type connectedClusters --cluster-name $azure_arc_k3s -g $k3s_rg_name -y
k get sa -n kube-system
k delete sa omsagent -n kube-system
k get sa -n kube-system
k get secrets -n kube-system
k delete secret omsagent-secret -n kube-system
k get cm -n kube-system
k delete cm omsagent-rs-config -n kube-system
k get cm -n kube-system
k delete deploy omsagent-rs -n kube-system
k get crd
k delete crd healthstates.azmon.container.insights
curl -o disable-monitoring.sh -L https://aka.ms/disable-monitoring-bash-script
bash disable-monitoring.sh --resource-id $azureArc_K3S_ClusterResourceId --kube-context $kubeContext
```
Then recreate :
```sh
az monitor log-analytics workspace create -n $analytics_workspace_name --location $location -g $k3s_rg_name --verbose
export analytics_workspace_id=$(az monitor log-analytics workspace show -n $analytics_workspace_name -g $k3s_rg_name -o tsv --query id)
echo "analytics_workspace_id:" $analytics_workspace_id
az k8s-extension create --name azuremonitor-containers --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings logAnalyticsWorkspaceResourceID=$analytics_workspace_id omsagent.resources.daemonset.limits.cpu=150m omsagent.resources.daemonset.limits.memory=600Mi omsagent.resources.deployment.limits.cpu=1 omsagent.resources.deployment.limits.memory=750Mi
az k8s-extension list --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters
azmon_extension_state=$(az k8s-extension show --name azuremonitor-containers --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters --query 'installState')
echo "Azure Monitor extension state: " $azmon_extension_state
```
Contributor guide
Assessment
This issue has not been assessed yet.