googleapis / googleapis/google-cloud-rust
use notification mechanism to block TokenCache refresh loop on exhaustive retries
- Dominant language
- Rust
- Stars
- 955
- Forks
- 144
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 279
Description
Currently, the `TokenCache` maintains valid access tokens via a background `refresh_task`. Transient errors (e.g. Metadata Server downtime) currently exhaust retries and abruptly terminate the background refresh task. Once the background task is dead, the cache emits outdated tokens or permanent failures, leaving the `Credential` struct poisoned. Users are forced to manually catch the failure and rebuild the entire authentication structure from scratch even during simple network outages.
This was reported on #4541.
In #4551 we will introduce a fix to the retry logic and the token cache loop to preserve the transient characteristic of some of these errors and introduce a backoff mechanism on the `refresh_task` instead of leaving the cache into an unusable state.
We've considered an alternative. Instead of sleeping, we can block the `refresh_task` using `tokio::sync::Notify` after a transient error and only wake it up when a new token request is made by the user. When a transient error occurs, the cache emits the error via the watch channel, and `refresh_task` calls `notify.notified().await` instead of sleeping. Any subsequent request to `TokenCache::latest_token_and_entity_tag()` (when a token is actively needed) would call `notify.notify_one()` to wake the background task and try again.
This seems the most complete solution and conscious on resource usage, but seemed prone to deadlocks. This issue is to capture this alternative and possibly implementing it in the future.
Contributor guide
Assessment
This issue has not been assessed yet.