Function CRD controller marks throttled commands as PermanentFailure without retry
- Dominant language
- Go
- Stars
- 28
- Forks
- 17
- Avg merge
- 11h 40m
- Merged PRs (30d)
- 42
Description
## Problem
When the adx-mon Helm chart is deployed and multiple `Function` CRDs are created simultaneously, the ADX cluster throttles some `.create-or-alter` commands with:
```
The control command was aborted due to throttling. Retrying after some backoff might succeed.
CommandType: 'DatabaseScriptExecute', Capacity: 20, Origin: 'RequestRateLimitPolicy/WorkloadGroup/default'.
```
The operator marks these as `PermanentFailure` and never retries them, leaving the ADX database in an incomplete state with missing functions.
## Observed behavior
On a fresh `taugrid-test` cluster (Sweden Central, dev SKU ADX), 9 Function CRDs were created at once. 5 of 9 failed with throttling and were marked `PermanentFailure`:
```
adx-mon-audit-events PermanentFailure
adx-mon-container-metrics PermanentFailure
adx-mon-experiment-metrics-dashboard-rows PermanentFailure
adx-mon-gpu-health PermanentFailure
adx-mon-nccl-errors PermanentFailure
adx-mon-kueue-metrics Success
adx-mon-node-health Success
adx-mon-training-job-summary Success
adx-mon-xid-errors PermanentFailure
```
The portal's Fleet page then fails with `SEM0260: Unknown function: 'GpuHealth'` because the function was never created in ADX.
## Expected behavior
- Throttle errors (HTTP 429 / capacity-exceeded) should be retried with exponential backoff, not treated as permanent failures.
- Only truly permanent errors (syntax errors, permission denied, database not found) should be marked `PermanentFailure`.
- Consider staggering function creation when multiple Function CRDs are reconciled simultaneously to avoid hitting rate limits in the first place.
## Workaround
Manually execute the `.create-or-alter` commands against the ADX endpoint:
```bash
# For each failed function CRD:
BODY=$(kubectl get functions.adx-mon.azure.com -n adx-mon -o jsonpath='{.spec.body}')
# Extract the .create-or-alter command and POST to ADX /v1/rest/mgmt
```
## Environment
- adx-mon chart version: deployed via TauGrid 0.3.2 Terraform
- ADX cluster SKU: Dev(No SLA)_Standard_E2a_v4 (Sweden Central)
- Kubernetes: AKS 1.35.7
- Function CRD count: 9 (across 3 databases: Metrics, Logs, Audit)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the Function CRD controller and the code that classifies ADX .create-or-alter command failures. Reproduce with multiple Function CRDs or the reported HTTP 429/capacity-exceeded response, then verify throttled commands retry with exponential backoff while syntax, permission, and missing-database errors remain permanent; completion should leave all functions created in ADX.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, helm, kubernetes
- Domain
- backend, databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100