googleapis / googleapis/google-cloud-java

[java-bigtable] Bigtable: BigtableCloudMonitoringExporter fails with "unrecognized metric labels" for connection_pool/outstanding_rpcs metric

Abierto
#13,059 2 comentarios 0 reacciones 0 asignados Ver en GitHub
api: bigtable
Lenguaje dominante
Java
Estrellas
2.1k
Forks
1.2k
Merge medio
1 d 23 h
PR fusionados (30 d)
154

Descripción

#### Environment details

1. Bigtable
2. Linux (GKE on Google Kubernetes Engine, us-central1)
3. OpenJDK 11
4. google-cloud-bigtable 2.73.0 (also reproducible with 2.69.0+)

#### Steps to reproduce

1. Create a `BigtableDataClient` with default settings (built-in metrics enabled by default)
2. Perform any Bigtable operation
3. Wait ~60 seconds for the metrics export cycle
4. Observe WARNING log from `BigtableCloudMonitoringExporter$1 onFailure`

#### Code example

```java
BigtableDataSettings settings = BigtableDataSettings.newBuilder()
.setProjectId("my-project")
.setInstanceId("my-instance")
.build();

BigtableDataClient client = BigtableDataClient.create(settings);

// Perform any operation, then wait ~60s for metrics export cycle
```

#### Stack trace

```
WARNING: createServiceTimeSeries request failed for application metrics.
shaded_com_google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client", resource.labels={"uuid": "java-32cd4f1a-8630-4708-b5d2-28f9d2f815cb1@mainstream-dequeue-6cb4c6d648-zjqr9-0", "cloud_platform": "gcp_kubernetes_engine", "host_name": "mainstream-dequeue-6cb4c6d648-zjqr9", "project_id": "dev-main-01", "region": "us-central1", "client_project": "dev-main-01", "app_profile": "", "instance": "analytics-data", "client_name": "java-bigtable/2.73.0", "host_id": "6238638445181415795"}): unrecognized metric labels [app_profile, project_id, client_name, instance]
at shaded_com_google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
at shaded_com_google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at shaded_com_google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
at shaded_com_google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125)
at shaded_com_google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
...
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client"): unrecognized metric labels [app_profile, project_id, client_name, instance]
at io.grpc.Status.asRuntimeException(Status.java:532)
... 18 more
```

#### External references such as API reference guides

- The `outstanding_rpcs` metric was introduced in v2.69.0 via https://github.com/googleapis/java-bigtable/pull/2696 and https://github.com/googleapis/java-bigtable/pull/2700

#### Any additional information below

The error occurs every ~60 seconds on every pod, across multiple pods simultaneously. The Cloud Monitoring metric descriptor for `bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs` does not recognize the labels `[app_profile, project_id, client_name, instance]` that the client sends. Either the server-side metric descriptor needs to be updated, or the client should not be sending these labels for this metric.

Bigtable data operations are unaffected — only client-side metrics export fails.

Workaround: disable built-in metrics with `BigtableDataSettings.newBuilder().setMetricsProvider(NoopMetricsProvider.create())`, but this disables all client-side Cloud Monitoring metrics.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.