googleapis / googleapis/google-cloud-java

[java-storage] Excessive ALTS handshake logs from gke-metadata-server when using Java GCS client with gRPC

Open
#12,633 4 comments 0 reactions 0 assignees View on GitHub
api: storage type: question
Dominant language
Java
Stars
2.1k
Forks
1.2k
Avg merge
1d 23h
Merged PRs (30d)
154

Description

### Description

We are observing an extremely high volume of informational logs (approximately 500,000 to 1,000,000 per day) from the `gke-metadata-server` container on our GKE cluster.

This appears to be triggered by our Java application that uses the `google-cloud-storage` library with the gRPC transport option to interact with Google Cloud Storage. Our application handles a high volume of requests, writing to GCS approximately **1 million times per day, with peaks of up to 1,000 writes per second**.

The excessive logging is causing a significant increase in costs for our monitoring and logging tool (Datadog).

The most frequent log messages are related to ALTS session handshakes:

```
[rpc-id:b0b2453b42e5f6b2] Closing ALTS session from pod "ml-test-common-historical-1".
[rpc-id:b0b2453b42e5f6b2] Use node identity as channel identity to connect to service: "storage.googleapis.com"
[rpc-id:b0b2453b42e5f6b2] Starting ALTS session from pod "ml-test-common-historical-1" running as test-common/ml-test-common-app to act as ml-test-common@evaluation-mila.iam.gserviceaccount.com
[rpc-id:b0b2453b42e5f6b2] Processing ALTS handshake request from "10.80.12.18:53498".
```

Given the high traffic, we suspect that the client is creating new ALTS sessions too frequently, rather than reusing existing ones, which results in this verbose logging.

### Environment

* **`google-cloud-storage` version:** `2.55.0`
* **Transport:** gRPC (`StorageOptions.grpc()`)
* **Environment:** Google Kubernetes Engine (GKE)
* **GKE Cluster Version:** `1.33.4-gke.1036000`
* **Workload Identity Enabled:** Yes
* **Java Version:** `24`

### Code Snippet

Our `Storage` client is initialized as a singleton and reused throughout the application's lifecycle. The following code demonstrates our usage pattern.

```java
// This class is a singleton
class MyClass {
final Storage storage;

public MyClass() {
// Client is initialized once using gRPC transport
this.storage = StorageOptions.grpc()
.setProjectId("your-gcp-project-id")
.build()
.getService();
}

@Override
public void saveData(String dataId, ByteBuffer data) throws Exception {
BlobInfo blobInfo = toBlobInfo(dataId);
Blob blob = this.storage.create(blobInfo, BlobTargetOption.doesNotExist());
try (WriteChannel wc = blob.writer()){
wc.write(data);
}
}
}
```

### Investigation So Far

We opened a support case with the Google Kubernetes Engine team. Their recommendation was to **"adjust the client library's connection pool to prevent frequent session creation."**

However, it is not clear from the `google-cloud-storage` or `google-api-grpc` documentation how to configure or tune the underlying gRPC connection pool for this purpose.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.