googleapis / googleapis/google-cloud-rust

use notification mechanism to block TokenCache refresh loop on exhaustive retries

Open
#4,593 0 comments 1 reaction 0 assignees View on GitHub
auth type: cleanup type: feature request
Dominant language
Rust
Stars
955
Forks
144
Avg merge
1d 5h
Merged PRs (30d)
279

Description

Currently, the `TokenCache` maintains valid access tokens via a background `refresh_task`. Transient errors (e.g. Metadata Server downtime) currently exhaust retries and abruptly terminate the background refresh task. Once the background task is dead, the cache emits outdated tokens or permanent failures, leaving the `Credential` struct poisoned. Users are forced to manually catch the failure and rebuild the entire authentication structure from scratch even during simple network outages.

This was reported on #4541.

In #4551 we will introduce a fix to the retry logic and the token cache loop to preserve the transient characteristic of some of these errors and introduce a backoff mechanism on the `refresh_task` instead of leaving the cache into an unusable state.

We've considered an alternative. Instead of sleeping, we can block the `refresh_task` using `tokio::sync::Notify` after a transient error and only wake it up when a new token request is made by the user. When a transient error occurs, the cache emits the error via the watch channel, and `refresh_task` calls `notify.notified().await` instead of sleeping. Any subsequent request to `TokenCache::latest_token_and_entity_tag()` (when a token is actively needed) would call `notify.notify_one()` to wake the background task and try again.

This seems the most complete solution and conscious on resource usage, but seemed prone to deadlocks. This issue is to capture this alternative and possibly implementing it in the future.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.