Operator stops sending heartbeat to hub ADX Cluster
- Dominant language
- Go
- Stars
- 28
- Forks
- 17
- Avg merge
- 11h 40m
- Merged PRs (30d)
- 42
Description
I have a set of kubernetes cluster operating in a spoke region that runs an ADX-Mon operator pod to send heartbeats to an ADX Hub in another region. It suddenly stopped sending heartbeats yesterday afternoon. Notice that the final heartbeat reconciliation cycle did not complete as it is missing the last 2 entries you'd expect.
```json
{"ts":"2026-04-05T17:28:29.457245Z","lvl":"INF","msg":"Reconciling"}
{"ts":"2026-04-05T17:28:29.457318Z","lvl":"INF","msg":"Reconciling ADXCluster adx-mon/adx-prod-eastus-1 (gene
ration 1)"}
{"ts":"2026-04-05T17:28:29.457332Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: executing partition clus
ter heartbeat"}
{"ts":"2026-04-05T17:28:29.457338Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: sending heartbeats to 1
federated clusters"}
{"ts":"2026-04-05T17:28:31.075749Z","lvl":"INF","msg":"Heartbeat sent to federated cluster https://adx-hub.fake.url"}
{"ts":"2026-04-05T17:28:31.075795Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: heartbeat cycle complete
, requeuing in 10m0s"}
...
{"ts":"2026-04-05T17:38:31.076158Z","lvl":"INF","msg":"Reconciling"}
{"ts":"2026-04-05T17:38:31.076243Z","lvl":"INF","msg":"Reconciling ADXCluster adx-mon/adx-prod-eastus-1 (gene
ration 1)"}
{"ts":"2026-04-05T17:38:31.076258Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: executing partition clus
ter heartbeat"}
{"ts":"2026-04-05T17:38:31.076265Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: sending heartbeats to 1
federated clusters"}
```
All logs in the operator are at level INF. Most of the other logs resemble this, where it appears to be logging watch operations.
```json
{"ts":"2026-04-02T17:50:41.038642Z","lvl":"INF","msg":"Watch close","values":["reflector","sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114","type","*v1.DaemonSet","totalItems",150]}
{"ts":"2026-04-02T17:50:41.038758Z","lvl":"INF","msg":"Request Body","values":["body",""]}
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the ADXCluster heartbeat reconciliation path described in the logs, comparing the completed cycle with the later cycle that stops after sending heartbeats. Review the controller-runtime informer entry point at sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114 and operator logs to identify the missing completion or requeue step. Done means heartbeats resume reliably and the failure is covered by a clear diagnostic or regression test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, go, kubernetes
- Domain
- distributed-systems, infrastructure, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100