Azure / Azure/AKS

[BUG] Prometheus containers located in ama-metrics-node and ama-metrics pods frequently restart

Open
#5,229 17 comments 0 reactions 1 assignee Assigned to @vishiy View on GitHub
addon/ama-metrics bug monitoring
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

### Describe the bug

I have a similar problem with [BUG] https://github.com/Azure/AKS/issues/3960 for newly installed AKS, but I also have ama-metrics-node pods errors.

Prometheus containers located in ama-metrics-node and ama-metrics pods are restarting almost every 20-25 minutes

### To Reproduce
Steps to reproduce the behavior:

Create the new private AKS on Azure
Create private dns for AKS + DNS for Azure Monitor
Create PrivateLink + Private Endpoint
See the problem (frequently restarting) in the kube-system namespace

### Expected behavior

They should not be restarted

### Environment

Private AKS Cluster
Kubernetes version: 1.32
PrivateDNS for AKS
PrivateDNS for Azure Monitor
PrivateLink + PrivateEndpoint

### Logs from the Pods

```
azmon-container-start-time file exists, reading start time
Container has been running for 7 minutes
{"level":"error","ts":"2025-08-24T19:57:17.389Z","caller":"internal/queue_sender.go:57","msg":"Exporting failed. Dropping data. Try enabling retry_on_failure config option to retry on retryable errors.","resource":{},"otelcol.component.id":"otlp","otelcol.component.kind":"exporter","otelcol.signal":"metrics","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:55680: connect: connection refused\"","dropped_items":3106,"stacktrace":"go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queue_sender.go:57\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queuebatch/disabled_batcher.go:22\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*asyncQueue[...]).Start.func1\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queuebatch/async_queue.go:47","filepath":"/opt/microsoft/otelcollector/collector-log.txt"}
{"level":"error","ts":"2025-08-24T19:57:20.995Z","caller":"internal/queue_sender.go:57","msg":"Exporting failed. Dropping data. Try enabling retry_on_failure config option to retry on retryable errors.","resource":{},"otelcol.component.id":"otlp","otelcol.component.kind":"exporter","otelcol.signal":"metrics","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:55680: connect: connection refused\"","dropped_items":157,"stacktrace":"go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queue_sender.go:57\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queuebatch/disabled_batcher.go:22\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*asyncQueue[...]).Start.func1\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queuebatch/async_queue.go:47","filepath":"/opt/microsoft/otelcollector/collector-log.txt"}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.