googleapis / googleapis/google-cloud-java

[java-storage] Excessive ALTS handshake logs from gke-metadata-server when using Java GCS client with gRPC

Offen
#12,633 4 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
api: storage type: question
Vorherrschende Sprache
Java
Sterne
2.1k
Forks
1.2k
Ø Merge
1 T. 23 Std.
Gemergte PRs (30 T.)
154

Beschreibung

### Description

We are observing an extremely high volume of informational logs (approximately 500,000 to 1,000,000 per day) from the `gke-metadata-server` container on our GKE cluster.

This appears to be triggered by our Java application that uses the `google-cloud-storage` library with the gRPC transport option to interact with Google Cloud Storage. Our application handles a high volume of requests, writing to GCS approximately **1 million times per day, with peaks of up to 1,000 writes per second**.

The excessive logging is causing a significant increase in costs for our monitoring and logging tool (Datadog).

The most frequent log messages are related to ALTS session handshakes:

```
[rpc-id:b0b2453b42e5f6b2] Closing ALTS session from pod "ml-test-common-historical-1".
[rpc-id:b0b2453b42e5f6b2] Use node identity as channel identity to connect to service: "storage.googleapis.com"
[rpc-id:b0b2453b42e5f6b2] Starting ALTS session from pod "ml-test-common-historical-1" running as test-common/ml-test-common-app to act as ml-test-common@evaluation-mila.iam.gserviceaccount.com
[rpc-id:b0b2453b42e5f6b2] Processing ALTS handshake request from "10.80.12.18:53498".
```

Given the high traffic, we suspect that the client is creating new ALTS sessions too frequently, rather than reusing existing ones, which results in this verbose logging.

### Environment

* **`google-cloud-storage` version:** `2.55.0`
* **Transport:** gRPC (`StorageOptions.grpc()`)
* **Environment:** Google Kubernetes Engine (GKE)
* **GKE Cluster Version:** `1.33.4-gke.1036000`
* **Workload Identity Enabled:** Yes
* **Java Version:** `24`

### Code Snippet

Our `Storage` client is initialized as a singleton and reused throughout the application's lifecycle. The following code demonstrates our usage pattern.

```java
// This class is a singleton
class MyClass {
final Storage storage;

public MyClass() {
// Client is initialized once using gRPC transport
this.storage = StorageOptions.grpc()
.setProjectId("your-gcp-project-id")
.build()
.getService();
}

@Override
public void saveData(String dataId, ByteBuffer data) throws Exception {
BlobInfo blobInfo = toBlobInfo(dataId);
Blob blob = this.storage.create(blobInfo, BlobTargetOption.doesNotExist());
try (WriteChannel wc = blob.writer()){
wc.write(data);
}
}
}
```

### Investigation So Far

We opened a support case with the Google Kubernetes Engine team. Their recommendation was to **"adjust the client library's connection pool to prevent frequent session creation."**

However, it is not clear from the `google-cloud-storage` or `google-api-grpc` documentation how to configure or tune the underlying gRPC connection pool for this purpose.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.