googleapis / googleapis/google-cloud-java

[java-bigtable] Bigtable: BigtableCloudMonitoringExporter fails with "unrecognized metric labels" for connection_pool/outstanding_rpcs metric

Đang mở
#13,059 2 bình luận 0 reaction 0 người được giao Xem trên GitHub
api: bigtable
Ngôn ngữ chính
Java
Star
2.1k
Fork
1.2k
Merge trung bình
1 ngày 23 giờ
Pull request đã merge (30 ngày)
154

Mô tả

#### Environment details

1. Bigtable
2. Linux (GKE on Google Kubernetes Engine, us-central1)
3. OpenJDK 11
4. google-cloud-bigtable 2.73.0 (also reproducible with 2.69.0+)

#### Steps to reproduce

1. Create a `BigtableDataClient` with default settings (built-in metrics enabled by default)
2. Perform any Bigtable operation
3. Wait ~60 seconds for the metrics export cycle
4. Observe WARNING log from `BigtableCloudMonitoringExporter$1 onFailure`

#### Code example

```java
BigtableDataSettings settings = BigtableDataSettings.newBuilder()
.setProjectId("my-project")
.setInstanceId("my-instance")
.build();

BigtableDataClient client = BigtableDataClient.create(settings);

// Perform any operation, then wait ~60s for metrics export cycle
```

#### Stack trace

```
WARNING: createServiceTimeSeries request failed for application metrics.
shaded_com_google.api.gax.rpc.InvalidArgumentException: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client", resource.labels={"uuid": "java-32cd4f1a-8630-4708-b5d2-28f9d2f815cb1@mainstream-dequeue-6cb4c6d648-zjqr9-0", "cloud_platform": "gcp_kubernetes_engine", "host_name": "mainstream-dequeue-6cb4c6d648-zjqr9", "project_id": "dev-main-01", "region": "us-central1", "client_project": "dev-main-01", "app_profile": "", "instance": "analytics-data", "client_name": "java-bigtable/2.73.0", "host_id": "6238638445181415795"}): unrecognized metric labels [app_profile, project_id, client_name, instance]
at shaded_com_google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:92)
at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:98)
at shaded_com_google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:66)
at shaded_com_google.api.gax.grpc.GrpcExceptionCallable$ExceptionTransformingFuture.onFailure(GrpcExceptionCallable.java:97)
at shaded_com_google.api.core.ApiFutures$1.onFailure(ApiFutures.java:84)
at shaded_com_google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1125)
at shaded_com_google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:30)
...
Caused by: io.grpc.StatusRuntimeException: INVALID_ARGUMENT: One or more TimeSeries could not be written: timeSeries[3] (metric.type="bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs", metric.labels={"app_profile": "", "project_id": "dev-main-01", "client_name": "bigtable-java/2.73.0", "instance": "analytics-data"}, resource.type="bigtable_client"): unrecognized metric labels [app_profile, project_id, client_name, instance]
at io.grpc.Status.asRuntimeException(Status.java:532)
... 18 more
```

#### External references such as API reference guides

- The `outstanding_rpcs` metric was introduced in v2.69.0 via https://github.com/googleapis/java-bigtable/pull/2696 and https://github.com/googleapis/java-bigtable/pull/2700

#### Any additional information below

The error occurs every ~60 seconds on every pod, across multiple pods simultaneously. The Cloud Monitoring metric descriptor for `bigtable.googleapis.com/internal/client/connection_pool/outstanding_rpcs` does not recognize the labels `[app_profile, project_id, client_name, instance]` that the client sends. Either the server-side metric descriptor needs to be updated, or the client should not be sending these labels for this metric.

Bigtable data operations are unaffected — only client-side metrics export fails.

Workaround: disable built-in metrics with `BigtableDataSettings.newBuilder().setMetricsProvider(NoopMetricsProvider.create())`, but this disables all client-side Cloud Monitoring metrics.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.