googleapis / googleapis/google-cloud-java

[java-bigtable] Bigtable: Use same project to write bigtable resource metrics and application resource metrics

Open
#13,068 2 comments 1 reaction 0 assignees View on GitHub
api: bigtable
Dominant language
Java
Stars
2.1k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
154

Description

#### Environment details
1. OS type and version: GCP / GKE
2. Java version: 21+
3. bigtable version(s): 2.47.0

#### Steps to reproduce

1. Deploy an application to GKE in a common project (e.g. mycompany-pro)
2. Use a custom project for Bigtable resources (e.g. mycompany-myapp-pro)
3. Use a service account that does have `monitoring.timeSeries.create` permissions on the custom project, but not on the common project
4. Enable client side metrics (enabled by default)

#### Code example

```java
// Create a default BigtableDataClient
BigtableDataSettings
.newBuilder()
.setProjectId(bigtableProperties.projectId)
.setInstanceId(bigtableProperties.instanceId)
.setAppProfileId(bigtableProperties.appProfileId)
```

#### Stack trace
```
createServiceTimeSeries request failed for bigtable metrics. Need monitoring metric writer permission on project= ... . Follow https://cloud.google.com/bigtable/docs/client-side-metrics-setup to set up permissions.
com.google.api.gax.rpc.PermissionDeniedException: io.grpc.StatusRuntimeException: PERMISSION_DENIED: Permission monitoring.timeSeries.create denied (or the resource may not exist).
```

#### External references such as API reference guides

- https://cloud.google.com/bigtable/docs/client-side-metrics-setup

#### Any additional information below

This seems to happen because in `BigtableCloudMonitoringExporter` different metrics are created for Bigtable resource metrics and what it calls `application resource metrics`. For Bigtable resource metrics, the Bigtable project is used and this is fine. However, the project for the application resource metrics is not customizable. The only metric it tries to record there seems to be `per_connection_error_count`. If the service account does not have the permission to create the time series in this common project, you receive the error above.

I'm not sure why the application resource metrics are needed for `per_connection_error_count` and why it defaults to using the project the application is running in instead of the pre-specified Bigtable project (which may be different).

Solving this could be done by either using the specified project, removing the application resource metrics or something else entirely.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.