Azure / Azure/adx-mon

Function CRD controller marks throttled commands as PermanentFailure without retry

Open
#1,238 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
28
Forks
17
Avg merge
11h 40m
Merged PRs (30d)
42

Description

## Problem

When the adx-mon Helm chart is deployed and multiple `Function` CRDs are created simultaneously, the ADX cluster throttles some `.create-or-alter` commands with:

```
The control command was aborted due to throttling. Retrying after some backoff might succeed.
CommandType: 'DatabaseScriptExecute', Capacity: 20, Origin: 'RequestRateLimitPolicy/WorkloadGroup/default'.
```

The operator marks these as `PermanentFailure` and never retries them, leaving the ADX database in an incomplete state with missing functions.

## Observed behavior

On a fresh `taugrid-test` cluster (Sweden Central, dev SKU ADX), 9 Function CRDs were created at once. 5 of 9 failed with throttling and were marked `PermanentFailure`:

```
adx-mon-audit-events PermanentFailure
adx-mon-container-metrics PermanentFailure
adx-mon-experiment-metrics-dashboard-rows PermanentFailure
adx-mon-gpu-health PermanentFailure
adx-mon-nccl-errors PermanentFailure
adx-mon-kueue-metrics Success
adx-mon-node-health Success
adx-mon-training-job-summary Success
adx-mon-xid-errors PermanentFailure
```

The portal's Fleet page then fails with `SEM0260: Unknown function: 'GpuHealth'` because the function was never created in ADX.

## Expected behavior

- Throttle errors (HTTP 429 / capacity-exceeded) should be retried with exponential backoff, not treated as permanent failures.
- Only truly permanent errors (syntax errors, permission denied, database not found) should be marked `PermanentFailure`.
- Consider staggering function creation when multiple Function CRDs are reconciled simultaneously to avoid hitting rate limits in the first place.

## Workaround

Manually execute the `.create-or-alter` commands against the ADX endpoint:

```bash
# For each failed function CRD:
BODY=$(kubectl get functions.adx-mon.azure.com -n adx-mon -o jsonpath='{.spec.body}')
# Extract the .create-or-alter command and POST to ADX /v1/rest/mgmt
```

## Environment

- adx-mon chart version: deployed via TauGrid 0.3.2 Terraform
- ADX cluster SKU: Dev(No SLA)_Standard_E2a_v4 (Sweden Central)
- Kubernetes: AKS 1.35.7
- Function CRD count: 9 (across 3 databases: Metrics, Logs, Audit)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the Function CRD controller and the code that classifies ADX .create-or-alter command failures. Reproduce with multiple Function CRDs or the reported HTTP 429/capacity-exceeded response, then verify throttled commands retry with exponential backoff while syntax, permission, and missing-database errors remain permanent; completion should leave all functions created in ADX.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, helm, kubernetes
Domain
backend, databases, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.