googleapis / googleapis/google-cloud-java

[java-bigtable] Bigtable: BigtableCloudMonitoringExporter fails with "unrecognized metric labels" for connection_pool/outstanding_rpcs metric

Open
#13,059 2 comments 0 reactions 0 assignees View on GitHub
api: bigtable
Dominant language
Java
Stars
2.1k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
154

Description

#### Environment details

1. Bigtable
2. Linux (GKE on Google Kubernetes Engine, us-central1)
3. OpenJDK 11
4. google-cloud-bigtable 2.73.0 (also reproducible with 2.69.0+)

#### Steps to reproduce

1. Create a `BigtableDataClient` with default settings (built-in metrics enabled by default)
2. Perform any Bigtable operation
3. Wait ~60 seconds for the metrics export cycle
4. Observe WARNING log from `BigtableCloudMonitoringExporter$1 onFailure`

#### Code example

```java
BigtableDataSettings settings = BigtableDataSettings.newBuilder()
.setProjectId("my-project")
.setInstanceId("my-instance")
.build();

BigtableDataClient client = BigtableDataClient.create(settings);

// Perform any operation, then wait ~60s for metrics export cycle
```

#### Stack trace

```
WARNING: createServiceTimeSeries request failed for application metrics.
shaded_com_google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client", resource.labels={"uuid": "java-32cd4f1a-8630-4708-b5d2-28f9d2f815cb1@mainstream-dequeue-6cb4c6d648-zjqr9-0", "cloud_platform": "gcp_kubernetes_engine", "host_name": "mainstream-dequeue-6cb4c6d648-zjqr9", "project_id": "dev-main-01", "region": "us-central1", "client_project": "dev-main-01", "app_profile": "", "instance": "analytics-data", "client_name": "java-bigtable/2.73.0", "host_id": "6238638445181415795"}): unrecognized metric labels [app_profile, project_id, client_name, instance]
at shaded_com_google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
at shaded_com_google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at shaded_com_google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
at shaded_com_google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125)
at shaded_com_google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
...
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client"): unrecognized metric labels [app_profile, project_id, client_name, instance]
at io.grpc.Status.asRuntimeException(Status.java:532)
... 18 more
```

#### External references such as API reference guides

- The `outstanding_rpcs` metric was introduced in v2.69.0 via https://github.com/googleapis/java-bigtable/pull/2696 and https://github.com/googleapis/java-bigtable/pull/2700

#### Any additional information below

The error occurs every ~60 seconds on every pod, across multiple pods simultaneously. The Cloud Monitoring metric descriptor for `bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs` does not recognize the labels `[app_profile, project_id, client_name, instance]` that the client sends. Either the server-side metric descriptor needs to be updated, or the client should not be sending these labels for this metric.

Bigtable data operations are unaffected — only client-side metrics export fails.

Workaround: disable built-in metrics with `BigtableDataSettings.newBuilder().setMetricsProvider(NoopMetricsProvider.create())`, but this disables all client-side Cloud Monitoring metrics.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.