agent-substrate / agent-substrate/substrate

[Bug]: Google Cloud Monitoring drops workqueue exponential histograms with INVALID_ARGUMENT (num_finite_buckets = 0)

オープン
#1,472 コメント 0 件 リアクション 0 件 担当者 1 名 @JeffLuoo が担当を希望しています GitHub で見る
area/observability bug-bash kind/bug prio/P2
主要言語
Go
スター
1.8k
フォーク
316
平均マージ
2日 43分
マージ済み PR(30日)
287

説明

### What happened?

When running Substrate on Google Kubernetes Engine (GKE) with the Managed OpenTelemetry addon enabled, `ate-controller` exports workqueue metrics through `controller-runtime` via `prombridge.NewMetricProducer`.

`controller-runtime` enables Prometheus native exponential histograms by default on `workqueue_work_duration_seconds` and `workqueue_queue_duration_seconds`. When a controller queue is idle or has not recorded any finite duration samples (such as `egressmitmtrust`), the OTLP exporter serializes an exponential histogram with `count: 0` and empty bucket arrays.

Every 60-second metric push tick, the GKE Managed OTel Collector (`otelcol-google` exporter) attempts to export these points to Google Cloud Monitoring. The Cloud Monitoring backend rejects the metric points with an `INVALID_ARGUMENT` gRPC error:

```
Field points[0].distributionValue had an invalid value: Distribution |bucket_options.exponential_buckets| has a |num_finite_buckets| of 0, which is less than 1.
```

The collector drops the data points and logs a `Partial success response` warning every minute.

### Expected Behavior

`ate-controller` metrics should be successfully ingested by Google Cloud Monitoring without data points being dropped or producing error spam in the collector logs.

### Steps to Reproduce

1. Deploy Agent Substrate on a GKE cluster with the Managed OpenTelemetry addon enabled:
```bash
./hack/install-ate.sh --deploy-ate-system
```
2. Check that `ate-controller` is running and exporting OTLP telemetry to `opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317`.
3. Inspect the logs of the Managed OTel collector:
```bash
kubectl logs -n gke-managed-otel -l app.kubernetes.io/name=opentelemetry-collector --tail=100 | grep "num_finite_buckets"
```
4. Observe the `INVALID_ARGUMENT` error and dropped data points every 60 seconds.

### Sandbox Runtime

Both / Runtime Agnostic

### Agent Substrate Version / Commit SHA

main

### Kubernetes Version & Environment

GKE 1.36.3

### Host OS & Architecture

Linux x86_64

### Relevant Logs and Diagnostic Output

```shell
{
"level": "warn",
"ts": "2026-09-04T14:05:16.912Z",
"caller": "otlpexporter/otlp.go:125",
"msg": "Partial success response",
"resource": {
"service.instance.id": "8ed03c03-6bad-47d7-b51f-19a633ddc8a5",
"service.name": "otelcol-google",
"service.version": "0.153.0-gke.2"
},
"otelcol.component.id": "otlp_grpc",
"otelcol.component.kind": "exporter",
"otelcol.signal": "metrics",
"message": "{\"errors\":[{\"code\":\"INVALID_ARGUMENT\",\"error_message\":\"Field points[0].distributionValue had an invalid value: Distribution |bucket_options.exponential_buckets| has a |num_finite_buckets| of 0, which is less than 1.\",\"num_data_points\":2,\"example_resource_attributes\":{\"k8s.namespace.name\":\"ate-system\",\"k8s.pod.name\":\"ate-controller-5888878bbb-pr985\",\"k8s.pod.uid\":\"3ea7a0e3-eeab-423e-92fa-f3ab1ea9c5d1\",\"service.instance.id\":\"3ea7a0e3-eeab-423e-92fa-f3ab1ea9c5d1\",\"service.name\":\"atecontroller\",\"telemetry.sdk.language\":\"go\",\"telemetry.sdk.name\":\"opentelemetry\",\"telemetry.sdk.version\":\"1.44.0\",\"k8s.pod.start_time\":\"2026-09-03T22:07:01Z\",\"k8s.replicaset.name\":\"ate-controller-5888878bbb\",\"k8s.deployment.name\":\"ate-controller\",\"k8s.node.name\":\"gke-otel-standard-cluste-default-pool-33a08656-6pa6\",\"gcp.project_id\":\"jeffluoo-gke-dev\",\"cloud.provider\":\"gcp\",\"cloud.platform\":\"gcp_kubernetes_engine\",\"cloud.account.id\":\"jeffluoo-gke-dev\",\"k8s.cluster.name\":\"otel-standard-cluster-2\",\"cloud.availability_zone\":\"us-central1-a\"},\"example_metric_name\":\"workqueue_work_duration_seconds\",\"example_metric_attributes\":{\"controller\":\"egressmitmtrust\",\"name\":\"egressmitmtrust\",\"top_level_controller_type\":\"Deployment\",\"top_level_controller_name\":\"ate-controller\"}}]}",
"dropped_data_points": 2
}
```

### Additional Context

_No response_

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [x] I have verified that this issue occurs on the latest commit on `main`.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。