agent-substrate / agent-substrate/substrate

[Bug]: Google Cloud Monitoring drops workqueue exponential histograms with INVALID_ARGUMENT (num_finite_buckets = 0)

Đang mở
#1,472 0 bình luận 0 reaction 1 người được giao Được @JeffLuoo nhận Xem trên GitHub
area/observability bug-bash kind/bug prio/P2
Ngôn ngữ chính
Go
Star
1.8k
Fork
316
Merge trung bình
2 ngày 43 phút
Pull request đã merge (30 ngày)
287

Mô tả

### What happened?

When running Substrate on Google Kubernetes Engine (GKE) with the Managed OpenTelemetry addon enabled, `ate-controller` exports workqueue metrics through `controller-runtime` via `prombridge.NewMetricProducer`.

`controller-runtime` enables Prometheus native exponential histograms by default on `workqueue_work_duration_seconds` and `workqueue_queue_duration_seconds`. When a controller queue is idle or has not recorded any finite duration samples (such as `egressmitmtrust`), the OTLP exporter serializes an exponential histogram with `count: 0` and empty bucket arrays.

Every 60-second metric push tick, the GKE Managed OTel Collector (`otelcol-google` exporter) attempts to export these points to Google Cloud Monitoring. The Cloud Monitoring backend rejects the metric points with an `INVALID_ARGUMENT` gRPC error:

```
Field points[0].distributionValue had an invalid value: Distribution |bucket_options.exponential_buckets| has a |num_finite_buckets| of 0, which is less than 1.
```

The collector drops the data points and logs a `Partial success response` warning every minute.

### Expected Behavior

`ate-controller` metrics should be successfully ingested by Google Cloud Monitoring without data points being dropped or producing error spam in the collector logs.

### Steps to Reproduce

1. Deploy Agent Substrate on a GKE cluster with the Managed OpenTelemetry addon enabled:
```bash
./hack/install-ate.sh --deploy-ate-system
```
2. Check that `ate-controller` is running and exporting OTLP telemetry to `opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317`.
3. Inspect the logs of the Managed OTel collector:
```bash
kubectl logs -n gke-managed-otel -l app.kubernetes.io/name=opentelemetry-collector --tail=100 | grep "num_finite_buckets"
```
4. Observe the `INVALID_ARGUMENT` error and dropped data points every 60 seconds.

### Sandbox Runtime

Both / Runtime Agnostic

### Agent Substrate Version / Commit SHA

main

### Kubernetes Version & Environment

GKE 1.36.3

### Host OS & Architecture

Linux x86_64

### Relevant Logs and Diagnostic Output

```shell
{
"level": "warn",
"ts": "2026-09-04T14:05:16.912Z",
"caller": "otlpexporter/otlp.go:125",
"msg": "Partial success response",
"resource": {
"service.instance.id": "8ed03c03-6bad-47d7-b51f-19a633ddc8a5",
"service.name": "otelcol-google",
"service.version": "0.153.0-gke.2"
},
"otelcol.component.id": "otlp_grpc",
"otelcol.component.kind": "exporter",
"otelcol.signal": "metrics",
"message": "{\"errors\":[{\"code\":\"INVALID_ARGUMENT\",\"error_message\":\"Field points[0].distributionValue had an invalid value: Distribution |bucket_options.exponential_buckets| has a |num_finite_buckets| of 0, which is less than 1.\",\"num_data_points\":2,\"example_resource_attributes\":{\"k8s.namespace.name\":\"ate-system\",\"k8s.pod.name\":\"ate-controller-5888878bbb-pr985\",\"k8s.pod.uid\":\"3ea7a0e3-eeab-423e-92fa-f3ab1ea9c5d1\",\"service.instance.id\":\"3ea7a0e3-eeab-423e-92fa-f3ab1ea9c5d1\",\"service.name\":\"atecontroller\",\"telemetry.sdk.language\":\"go\",\"telemetry.sdk.name\":\"opentelemetry\",\"telemetry.sdk.version\":\"1.44.0\",\"k8s.pod.start_time\":\"2026-09-03T22:07:01Z\",\"k8s.replicaset.name\":\"ate-controller-5888878bbb\",\"k8s.deployment.name\":\"ate-controller\",\"k8s.node.name\":\"gke-otel-standard-cluste-default-pool-33a08656-6pa6\",\"gcp.project_id\":\"jeffluoo-gke-dev\",\"cloud.provider\":\"gcp\",\"cloud.platform\":\"gcp_kubernetes_engine\",\"cloud.account.id\":\"jeffluoo-gke-dev\",\"k8s.cluster.name\":\"otel-standard-cluster-2\",\"cloud.availability_zone\":\"us-central1-a\"},\"example_metric_name\":\"workqueue_work_duration_seconds\",\"example_metric_attributes\":{\"controller\":\"egressmitmtrust\",\"name\":\"egressmitmtrust\",\"top_level_controller_type\":\"Deployment\",\"top_level_controller_name\":\"ate-controller\"}}]}",
"dropped_data_points": 2
}
```

### Additional Context

_No response_

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [x] I have verified that this issue occurs on the latest commit on `main`.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.