Azure / Azure/adx-mon

Operator stops sending heartbeat to hub ADX Cluster

Open
#1,104 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
28
Forks
17
Avg merge
11h 40m
Merged PRs (30d)
42

Description

I have a set of kubernetes cluster operating in a spoke region that runs an ADX-Mon operator pod to send heartbeats to an ADX Hub in another region. It suddenly stopped sending heartbeats yesterday afternoon. Notice that the final heartbeat reconciliation cycle did not complete as it is missing the last 2 entries you'd expect.

```json
{"ts":"2026-04-05T17:28:29.457245Z","lvl":"INF","msg":"Reconciling"}
{"ts":"2026-04-05T17:28:29.457318Z","lvl":"INF","msg":"Reconciling ADXCluster adx-mon/adx-prod-eastus-1 (gene
ration 1)"}
{"ts":"2026-04-05T17:28:29.457332Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: executing partition clus
ter heartbeat"}
{"ts":"2026-04-05T17:28:29.457338Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: sending heartbeats to 1
federated clusters"}
{"ts":"2026-04-05T17:28:31.075749Z","lvl":"INF","msg":"Heartbeat sent to federated cluster https://adx-hub.fake.url"}
{"ts":"2026-04-05T17:28:31.075795Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: heartbeat cycle complete
, requeuing in 10m0s"}
...
{"ts":"2026-04-05T17:38:31.076158Z","lvl":"INF","msg":"Reconciling"}
{"ts":"2026-04-05T17:38:31.076243Z","lvl":"INF","msg":"Reconciling ADXCluster adx-mon/adx-prod-eastus-1 (gene
ration 1)"}
{"ts":"2026-04-05T17:38:31.076258Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: executing partition clus
ter heartbeat"}
{"ts":"2026-04-05T17:38:31.076265Z","lvl":"INF","msg":"ADXCluster adx-prod-eastus-1: sending heartbeats to 1
federated clusters"}
```

All logs in the operator are at level INF. Most of the other logs resemble this, where it appears to be logging watch operations.

```json
{"ts":"2026-04-02T17:50:41.038642Z","lvl":"INF","msg":"Watch close","values":["reflector","sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114","type","*v1.DaemonSet","totalItems",150]}
{"ts":"2026-04-02T17:50:41.038758Z","lvl":"INF","msg":"Request Body","values":["body",""]}
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the ADXCluster heartbeat reconciliation path described in the logs, comparing the completed cycle with the later cycle that stops after sending heartbeats. Review the controller-runtime informer entry point at sigs.k8s.io/controller-runtime/pkg/cache/internal/informers.go:114 and operator logs to identify the missing completion or requeue step. Done means heartbeats resume reliably and the failure is covered by a clear diagnostic or regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, go, kubernetes
Domain
distributed-systems, infrastructure, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.