googleapis / googleapis/google-cloud-php

[Auth]: Failed to read new token file after symlink rotation

Open
#9,674 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
PHP
Stars
1.2k
Forks
463
Avg merge
2d 1h
Merged PRs (30d)
145

Description

This is a client library issue. The product (STS token exchange) behaves correctly. The failure is in how `Google\Auth\CredentialSource\FileSource` reads the subject token file.

#### Environment details

- Programming language: PHP
- OS: Linux (Debian-based container) on Amazon EKS
- Language runtime version: PHP 8.4 (default `realpath_cache_ttl` = 120, `realpath_cache_size` = 4096K)
- Package version: `google/auth` v1.50.0. The code is unchanged in v1.53.0:
https://github.com/googleapis/google-auth-library-php/blob/v1.53.0/src/CredentialSource/FileSource.php

#### Summary

We use Workload Identity Federation from EKS. The Kubernetes service account token is mounted as a projected volume and referenced from the credential config with `credential_source.file`.

Kubelet rotates a projected token with an atomic symlink swap: it writes the new token into a new `..YYYY_MM_DD_...` directory, repoints the `..data` symlink, and deletes the old directory. PHP caches the resolved path of `credential_source.file` in its realpath cache for up to `realpath_cache_ttl` seconds (120 by default). After the swap, `file_get_contents()` on the unchanged path resolves to the deleted directory and fails.

`FileSource::fetchSubjectToken()` does a single `file_get_contents($this->file)` with no `clearstatcache()` and no retry. Every token refresh in that window fails with:

```
file_get_contents(/var/run/secrets/tokens/eks-token): Failed to open stream: No such file or directory
```

In comparison, the AWS SDK for PHP handles the same event explicitly for IRSA. On a failed read it clears the stat cache for the symlink target, the target directory, and the path, then re-reads with bounded retries:
https://github.com/aws/aws-sdk-php/blob/3.374.2/src/Credentials/AssumeRoleWithWebIdentityCredentialProvider.php#L103-L114

This is why the AWS token on the same pods does not have the problem.

#### Steps to reproduce

1. Run a long-lived PHP 8.4 process (for example PHP-FPM) on Kubernetes with a projected service account token volume and a WIF credential config that points `credential_source.file` at it. Set a short `expirationSeconds` (for example 600) so kubelet rotates the token often.
2. Configure `ExternalAccountCredentials` (or `ApplicationDefaultCredentials` with `GOOGLE_APPLICATION_CREDENTIALS` set to the config) and make requests continuously so the credential is refreshed.
3. Wait for kubelet to rotate the token. Kubelet rotates when the token passes 80% of its lifetime or 24 hours, whichever comes first.
4. Observe `fetchSubjectToken()` fail with `No such file or directory` for up to 120 seconds after the rotation, until the realpath cache entry expires.

A unit-level reproduction that does not need Kubernetes: read a file through a symlink once with `file_get_contents()`, replace the symlink target with a new directory, delete the old directory, and read the same path again inside the same process without calling `clearstatcache()`.

#### Expected behaviour

`FileSource` reads the current token after a rotation. A single failed read of a path that exists should not surface to the caller as an authentication failure.

#### Proposed fix

Two changes would resolve this. Either one alone would help.

1. Make `FileSource::fetchSubjectToken()` resilient to the swap. On a failed read: `readlink()` the path, `clearstatcache(true, ...)` on the resolved target, its parent directory, and the path itself, then re-read. Retry a bounded number of times with backoff when the read still fails or returns an empty string. This mirrors the AWS SDK implementation linked above.

2. Allow callers to supply their own `ExternalAccountCredentialSourceInterface`. Today `ExternalAccountCredentials::__construct()` calls the private `buildCredentialSource()` which always does `new FileSource(...)`. There is no constructor option, setter, or factory hook. We had to use reflection to swap the private `subjectTokenFetcher` property, which is fragile:
https://github.com/googleapis/google-auth-library-php/blob/v1.53.0/src/Credentials/ExternalAccountCredentials.php

#### Workaround we use today

We implemented an `ExternalAccountCredentialSourceInterface` that clears the stat cache and retries, and injected it via reflection into `ExternalAccountCredentials`. We also raised the token `expirationSeconds` to 86400 to limit rotations to one per day per pod. This reduces exposure but does not remove the race.

We are happy to open a pull request for either or both of the proposed changes if maintainers agree with the approach.

Contributor guide

Open the contributing guide

Research direction

Start with src/CredentialSource/FileSource.php and its fetchSubjectToken() path, then inspect src/Credentials/ExternalAccountCredentials.php and the AWS SDK retry behavior linked in the issue. Reproduce the failure by rotating a symlink target within one PHP process, and consider how to verify that the current token is read successfully after rotation without exposing a transient authentication failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
php
Domain
authentication
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.