agent-substrate / agent-substrate/substrate

[Bug]: Google Cloud Monitoring drops workqueue exponential histograms with INVALID_ARGUMENT (num_finite_buckets = 0)

Abierto
#1,472 0 comentarios 0 reacciones 1 asignado Reclamado por @JeffLuoo Ver en GitHub
area/observability bug-bash kind/bug prio/P2
Lenguaje dominante
Go
Estrellas
1.8k
Forks
316
Merge medio
2 d 43 min
PR fusionados (30 d)
287

Descripción

### What happened?

When running Substrate on Google Kubernetes Engine (GKE) with the Managed OpenTelemetry addon enabled, `ate-controller` exports workqueue metrics through `controller-runtime` via `prombridge.NewMetricProducer`.

`controller-runtime` enables Prometheus native exponential histograms by default on `workqueue_work_duration_seconds` and `workqueue_queue_duration_seconds`. When a controller queue is idle or has not recorded any finite duration samples (such as `egressmitmtrust`), the OTLP exporter serializes an exponential histogram with `count: 0` and empty bucket arrays.

Every 60-second metric push tick, the GKE Managed OTel Collector (`otelcol-google` exporter) attempts to export these points to Google Cloud Monitoring. The Cloud Monitoring backend rejects the metric points with an `INVALID_ARGUMENT` gRPC error:

```
Field points[0].distributionValue had an invalid value: Distribution |bucket_options.exponential_buckets| has a |num_finite_buckets| of 0, which is less than 1.
```

The collector drops the data points and logs a `Partial success response` warning every minute.

### Expected Behavior

`ate-controller` metrics should be successfully ingested by Google Cloud Monitoring without data points being dropped or producing error spam in the collector logs.

### Steps to Reproduce

1. Deploy Agent Substrate on a GKE cluster with the Managed OpenTelemetry addon enabled:
```bash
./hack/install-ate.sh --deploy-ate-system
```
2. Check that `ate-controller` is running and exporting OTLP telemetry to `opentelemetry-collector.gke-managed-otel.svc.cluster.local:4317`.
3. Inspect the logs of the Managed OTel collector:
```bash
kubectl logs -n gke-managed-otel -l app.kubernetes.io/name=opentelemetry-collector --tail=100 | grep "num_finite_buckets"
```
4. Observe the `INVALID_ARGUMENT` error and dropped data points every 60 seconds.

### Sandbox Runtime

Both / Runtime Agnostic

### Agent Substrate Version / Commit SHA

main

### Kubernetes Version & Environment

GKE 1.36.3

### Host OS & Architecture

Linux x86_64

### Relevant Logs and Diagnostic Output

```shell
{
"level": "warn",
"ts": "2026-09-04T14:05:16.912Z",
"caller": "otlpexporter/otlp.go:125",
"msg": "Partial success response",
"resource": {
"service.instance.id": "8ed03c03-6bad-47d7-b51f-19a633ddc8a5",
"service.name": "otelcol-google",
"service.version": "0.153.0-gke.2"
},
"otelcol.component.id": "otlp_grpc",
"otelcol.component.kind": "exporter",
"otelcol.signal": "metrics",
"message": "{\"errors\":[{\"code\":\"INVALID_ARGUMENT\",\"error_message\":\"Field points[0].distributionValue had an invalid value: Distribution |bucket_options.exponential_buckets| has a |num_finite_buckets| of 0, which is less than 1.\",\"num_data_points\":2,\"example_resource_attributes\":{\"k8s.namespace.name\":\"ate-system\",\"k8s.pod.name\":\"ate-controller-5888878bbb-pr985\",\"k8s.pod.uid\":\"3ea7a0e3-eeab-423e-92fa-f3ab1ea9c5d1\",\"service.instance.id\":\"3ea7a0e3-eeab-423e-92fa-f3ab1ea9c5d1\",\"service.name\":\"atecontroller\",\"telemetry.sdk.language\":\"go\",\"telemetry.sdk.name\":\"opentelemetry\",\"telemetry.sdk.version\":\"1.44.0\",\"k8s.pod.start_time\":\"2026-09-03T22:07:01Z\",\"k8s.replicaset.name\":\"ate-controller-5888878bbb\",\"k8s.deployment.name\":\"ate-controller\",\"k8s.node.name\":\"gke-otel-standard-cluste-default-pool-33a08656-6pa6\",\"gcp.project_id\":\"jeffluoo-gke-dev\",\"cloud.provider\":\"gcp\",\"cloud.platform\":\"gcp_kubernetes_engine\",\"cloud.account.id\":\"jeffluoo-gke-dev\",\"k8s.cluster.name\":\"otel-standard-cluster-2\",\"cloud.availability_zone\":\"us-central1-a\"},\"example_metric_name\":\"workqueue_work_duration_seconds\",\"example_metric_attributes\":{\"controller\":\"egressmitmtrust\",\"name\":\"egressmitmtrust\",\"top_level_controller_type\":\"Deployment\",\"top_level_controller_name\":\"ate-controller\"}}]}",
"dropped_data_points": 2
}
```

### Additional Context

_No response_

### Confirmation

- [x] I have searched existing issues and verified that this is not a duplicate.
- [x] I have verified that this issue occurs on the latest commit on `main`.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.