googleapis / googleapis/google-cloud-java
[java-bigtable] Bigtable: BigtableCloudMonitoringExporter fails with "unrecognized metric labels" for connection_pool/outstanding_rpcs metric
- Langage dominant
- Java
- Étoiles
- 2.1k
- Forks
- 1.2k
- Merge moyen
- 1 j 23 h
- PR mergées (30 j)
- 154
Description
#### Environment details
1. Bigtable
2. Linux (GKE on Google Kubernetes Engine, us-central1)
3. OpenJDK 11
4. google-cloud-bigtable 2.73.0 (also reproducible with 2.69.0+)
#### Steps to reproduce
1. Create a `BigtableDataClient` with default settings (built-in metrics enabled by default)
2. Perform any Bigtable operation
3. Wait ~60 seconds for the metrics export cycle
4. Observe WARNING log from `BigtableCloudMonitoringExporter$1 onFailure`
#### Code example
```java
BigtableDataSettings settings = BigtableDataSettings.newBuilder()
.setProjectId("my-project")
.setInstanceId("my-instance")
.build();
BigtableDataClient client = BigtableDataClient.create(settings);
// Perform any operation, then wait ~60s for metrics export cycle
```
#### Stack trace
```
WARNING: createServiceTimeSeries request failed for application metrics.
shaded_com_google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client", resource.labels={"uuid": "java-32cd4f1a-8630-4708-b5d2-28f9d2f815cb1@mainstream-dequeue-6cb4c6d648-zjqr9-0", "cloud_platform": "gcp_kubernetes_engine", "host_name": "mainstream-dequeue-6cb4c6d648-zjqr9", "project_id": "dev-main-01", "region": "us-central1", "client_project": "dev-main-01", "app_profile": "", "instance": "analytics-data", "client_name": "java-bigtable/2.73.0", "host_id": "6238638445181415795"}): unrecognized metric labels [app_profile, project_id, client_name, instance]
at shaded_com_google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
at shaded_com_google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at shaded_com_google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
at shaded_com_google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125)
at shaded_com_google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
...
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client"): unrecognized metric labels [app_profile, project_id, client_name, instance]
at io.grpc.Status.asRuntimeException(Status.java:532)
... 18 more
```
#### External references such as API reference guides
- The `outstanding_rpcs` metric was introduced in v2.69.0 via https://github.com/googleapis/java-bigtable/pull/2696 and https://github.com/googleapis/java-bigtable/pull/2700
#### Any additional information below
The error occurs every ~60 seconds on every pod, across multiple pods simultaneously. The Cloud Monitoring metric descriptor for `bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs` does not recognize the labels `[app_profile, project_id, client_name, instance]` that the client sends. Either the server-side metric descriptor needs to be updated, or the client should not be sending these labels for this metric.
Bigtable data operations are unaffected — only client-side metrics export fails.
Workaround: disable built-in metrics with `BigtableDataSettings.newBuilder().setMetricsProvider(NoopMetricsProvider.create())`, but this disables all client-side Cloud Monitoring metrics.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.