Azure / Azure/aksArc

Known issue - AKS Arc billing status goes out-of-policy - can be fixed without re-installation

Open
#372 0 comments 0 reactions 0 assignees View on GitHub
Document/ TSG
Dominant language
PowerShell
Stars
118
Forks
63
PR merge metrics
No merged PRs in 30d

Description

An AKS Arc cluster billing status had been out of sync and limited cluster operations.
![image](https://github.com/Azure/aksArc/assets/6702311/f4a2e366-8e1c-40a5-b199-e09344817070)

`Sync-AksHciBilling` returns ` Upload Successful!` but ` Get-AksHciBillingStatus | ConvertTo-Json` shows a ConnectionStatus `OutOfPolicy` and the `LastConnected` date is higher than 30 days.

It is documented in the [known issues](https://learn.microsoft.com/en-us/azure/aks/hybrid/known-issues-upgrade#aks-arc-goes-out-of-policy-if-a-workload-cluster-hasn-t-been-created-in-60-days) with the only way to fix as a redeploy with a clean installation.

With help from support, we fixed the issue by deleting the billing-manager pods.
`kubectl get pods -n azure-arc --kubeconfig (get-kvaconfig).kubeconfig` showed the billing-manager deployment older than 76 days
Billing_manager_hangs

- Deleting the pods `kubectl delete pod -n azure-arc billing-manager-deployment-597b8997bb-kzksb --kubeconfig (get-kvaconfig).kubeconfig`
- `Sync-AksHciBilling`
- Fixed the issue and `Get-AksHciBillingStatus | ConvertTo-Json` reported ConnectionStatus `Connected` again.

We created multiple workload clusters after initial setup, but did not touch them for more than 60 days.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the documented known issue and reproducing the reported commands: Get-AksHciBillingStatus, Sync-AksHciBilling, and kubectl inspection of the billing-manager pods in the azure-arc namespace. Compare behavior before and after the billing-manager pods are deleted. Done means the documented recovery path no longer requires a clean redeployment and billing status returns to Connected after synchronization.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, powershell
Domain
cloud, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.