Azure / Azure/azure-cli-extensions

Az Monitor for Arc k8s extension deletion/reinstall fails

Open
#3,315 2 comments 0 reactions 0 assignees View on GitHub
Kubernetes Configuration Service Attention
Dominant language
Python
Stars
454
Forks
1.7k
Avg merge
2d 19h
Merged PRs (30d)
64

Description

- If the issue is to do with Azure CLI 2.0 in-particular, create an issue here at [Azure/azure-cli](https://github.com/Azure/azure-cli/issues)

### Extension name (the extension in question)
k8s-extension

### Description of issue (in as much detail as possible)
az k8s-extension delete does not delete rsources from the cluster

The extnesion is displayed as FAILEd then in the Azure portal with errors :

```console
Error: {failed to install chart from path [] for release [azuremonitor-containers]: err [rendered manifests contain a resource that already exists. Unable to continue with install: ServiceAccount "omsagent" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "azuremonitor-containers": current value is "azmon-containers-release-1"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "azuremonitor-containers": current value is "default"]} occurred while doing the operation : {Installing the extension} on the config
```

```console
Error: {failed to install chart from path [] for release [azuremonitor-containers]: err [rendered manifests contain a resource that already exists. Unable to continue with install: Secret "omsagent-secret" in namespace "kube-system" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "azuremonitor-containers": current value is "azmon-containers-release-1"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "azuremonitor-containers": current value is "default"]} occurred while doing the operation : {Installing the extension} on the config
```

-----
```sh
az version
```

```console
{
"azure-cli": "2.22.1",
"azure-cli-core": "2.22.1",
"azure-cli-telemetry": "1.0.6",
"extensions": {
"connectedk8s": "1.1.3",
"connectedmachine": "0.3.0",
"k8s-configuration": "1.0.0",
"k8s-extension": "0.2.1",
"spring-cloud": "2.3.0"
}
}
```

I di test it on a K3S cluster deployed on an Azure VM

workaround : I had to manually clean-up the environment :
```sh
az k8s-extension delete --name azuremonitor-containers --cluster-type connectedClusters --cluster-name $azure_arc_k3s -g $k3s_rg_name -y
k get sa -n kube-system
k delete sa omsagent -n kube-system
k get sa -n kube-system
k get secrets -n kube-system
k delete secret omsagent-secret -n kube-system
k get cm -n kube-system
k delete cm omsagent-rs-config -n kube-system
k get cm -n kube-system
k delete deploy omsagent-rs -n kube-system
k get crd
k delete crd healthstates.azmon.container.insights

curl -o disable-monitoring.sh -L https://aka.ms/disable-monitoring-bash-script
bash disable-monitoring.sh --resource-id $azureArc_K3S_ClusterResourceId --kube-context $kubeContext

```

Then recreate :
```sh
az monitor log-analytics workspace create -n $analytics_workspace_name --location $location -g $k3s_rg_name --verbose
export analytics_workspace_id=$(az monitor log-analytics workspace show -n $analytics_workspace_name -g $k3s_rg_name -o tsv --query id)
echo "analytics_workspace_id:" $analytics_workspace_id

az k8s-extension create --name azuremonitor-containers --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters --extension-type Microsoft.AzureMonitor.Containers --configuration-settings logAnalyticsWorkspaceResourceID=$analytics_workspace_id omsagent.resources.daemonset.limits.cpu=150m omsagent.resources.daemonset.limits.memory=600Mi omsagent.resources.deployment.limits.cpu=1 omsagent.resources.deployment.limits.memory=750Mi

az k8s-extension list --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters
azmon_extension_state=$(az k8s-extension show --name azuremonitor-containers --cluster-name $azure_arc_k3s --resource-group $k3s_rg_name --cluster-type connectedClusters --query 'installState')
echo "Azure Monitor extension state: " $azmon_extension_state
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.