[BUG] Prometheus containers located in ama-metrics-node and ama-metrics pods frequently restart
- Dominant language
- TypeScript
- Stars
- 2.1k
- Forks
- 395
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 13
Description
### Describe the bug
I have a similar problem with [BUG] https://github.com/Azure/AKS/issues/3960 for newly installed AKS, but I also have ama-metrics-node pods errors.
Prometheus containers located in ama-metrics-node and ama-metrics pods are restarting almost every 20-25 minutes
### To Reproduce
Steps to reproduce the behavior:
Create the new private AKS on Azure
Create private dns for AKS + DNS for Azure Monitor
Create PrivateLink + Private Endpoint
See the problem (frequently restarting) in the kube-system namespace
### Expected behavior
They should not be restarted
### Environment
Private AKS Cluster
Kubernetes version: 1.32
PrivateDNS for AKS
PrivateDNS for Azure Monitor
PrivateLink + PrivateEndpoint
### Logs from the Pods
```
azmon-container-start-time file exists, reading start time
Container has been running for 7 minutes
{"level":"error","ts":"2025-08-24T19:57:17.389Z","caller":"internal/queue_sender.go:57","msg":"Exporting failed. Dropping data. Try enabling retry_on_failure config option to retry on retryable errors.","resource":{},"otelcol.component.id":"otlp","otelcol.component.kind":"exporter","otelcol.signal":"metrics","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:55680: connect: connection refused\"","dropped_items":3106,"stacktrace":"go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queue_sender.go:57\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queuebatch/disabled_batcher.go:22\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*asyncQueue[...]).Start.func1\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queuebatch/async_queue.go:47","filepath":"/opt/microsoft/otelcollector/collector-log.txt"}
{"level":"error","ts":"2025-08-24T19:57:20.995Z","caller":"internal/queue_sender.go:57","msg":"Exporting failed. Dropping data. Try enabling retry_on_failure config option to retry on retryable errors.","resource":{},"otelcol.component.id":"otlp","otelcol.component.kind":"exporter","otelcol.signal":"metrics","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing: dial tcp 127.0.0.1:55680: connect: connection refused\"","dropped_items":157,"stacktrace":"go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queue_sender.go:57\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queuebatch/disabled_batcher.go:22\ngo.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*asyncQueue[...]).Start.func1\n\t/go/pkg/mod/go.opentelemetry.io/collector/exporter@v0.127.0/exporterhelper/internal/queuebatch/async_queue.go:47","filepath":"/opt/microsoft/otelcollector/collector-log.txt"}
```
Contributor guide
Assessment
This issue has not been assessed yet.