Azure / Azure/taugrid

adx-mon: retry Function reconciliation after transient ADX throttling

Open
#162 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
15
Forks
3
Avg merge
21h 53m
Merged PRs (30d)
131

Description

## Summary

A fresh ADX deployment can leave all adx-mon Function custom resources in `PermanentFailure` after initial ADX control plane throttling. This prevents creation of `GpuHealth()` even though raw GPU telemetry is successfully ingested.

## Reproduction evidence

- adx-mon was installed with its pre-built Function CRs enabled.
- Raw DCGM metric tables were created and queried successfully. `DCGMFIDEVGPUUTIL` contained fresh samples from the cluster.
- The `adx-mon-gpu-health` Function CR had `Reconciled=False` and `status=PermanentFailure`.
- Its status error was an ADX control plane throttle:

```
The control command was aborted due to throttling. Retrying after some backoff might succeed.
CommandType: DatabaseScriptExecute
Capacity: 20
Origin: RequestRateLimitPolicy/WorkloadGroup/default
```

- Other Function CRs showed the same failure mode.
- The controller did not reconcile them again after the throttling window passed.
- The ADX Metrics database consequently had no `GpuHealth()` function.

## Impact

The Portal can show Kusto-backed runs and Kubernetes node inventory, but its GPU Health and Cost boards return HTTP 502 because their KQL calls `GpuHealth()` and ADX reports `Unknown function: 'GpuHealth'`.

This is not a DCGM exporter ingestion issue and not a Portal query bug. The raw data path is healthy; the derived Kusto function reconciliation is stuck after a transient failure.

## Proposed fix

1. Classify ADX throttling and other explicitly transient ADX control plane responses as retryable.
2. Requeue Function reconciliation with bounded exponential backoff and jitter rather than setting terminal `PermanentFailure`.
3. Reserve `PermanentFailure` for deterministic errors such as invalid KQL or incompatible schema.
4. Expose a clear retrying versus terminal status condition and reason in the Function CR.
5. Consider pacing or batching initial Function creation to reduce the initial control plane burst.
6. Ensure Helm or an operator readiness check can wait for required Function CRs to report `Reconciled=True`.

## Acceptance criteria

- Simulated or live ADX 429 throttling causes Function CRs to requeue and eventually reconcile.
- `GpuHealth()` appears after the throttle window without deleting or manually editing the CR.
- A Portal GPU Health query succeeds after function reconciliation when raw DCGM tables contain data.
- Invalid KQL remains a terminal, actionable failure and does not retry indefinitely.

Contributor guide

Open the contributing guide

Research direction

Start at the Function custom-resource controller's reconciliation path and inspect how ADX control-plane errors become PermanentFailure; the issue names no files or tests. Exercise the reconciliation with a throttling response and verify that transient failures requeue while invalid KQL remains terminal. Done means functions eventually reconcile, the retrying and terminal conditions are clear, and GpuHealth() becomes available without manual CR changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, helm, kubernetes
Domain
backend, cloud, devops
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.