apache / apache/iceberg

REST OAuth2 session never recovers after a failed token refresh

Open
#17,756 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

### Apache Iceberg version

1.11.0 (latest release)

### Query engine

None

### Please describe the bug 🐞

One failed background OAuth2 token refresh permanently stops refresh for that `AuthSession`.

After the existing token expires, REST catalog requests continue using it and receive 401 responses for the remaining lifetime of the catalog object. Restoring the token endpoint does not recover the session.

I reproduced this against Iceberg 1.10.0 with a standalone HTTP stub. The same control flow is present
in 1.11.0 and on `main`:

https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java

### Cause

Token refresh is implemented as a chain of one-shot scheduled tasks. Each successful refresh schedules the next task:

```java
Pair expiration = session.refresh(client);
if (expiration != null) {
scheduleTokenRefresh(
client,
executor,
session,
refreshStartTime + expiration.second().toMillis(expiration.first()));
}
```

`AuthSession.refresh` returns null after its refresh attempts and credential fallback fail. The scheduled task then returns without scheduling another task. No manager or request path re-arms the catalog session:

- `authenticate` continues attaching the token already stored in the session.
- `OAuth2Manager.contextualSession` continues using or deriving from the existing catalog session.
- A 401 is mapped to `NotAuthorizedException` and does not initiate token renewal.

The current token is normally still valid when the background refresh fails, so the visible 401s
begin later, after the refresh warning is no longer near the failing request in the log.

### Retry window

The refresh uses `Tasks.retry(5)`, which gives six attempts. With the configured exponential backoff,
the sleeps are approximately 100, 200, 400, 800, and 1600 milliseconds. That is about 3.1 seconds of
sleep before the attempt count is exhausted. Request duration adds to that time.

`onFailure` then attempts credential-based recovery once. If that also fails, refresh returns null and
the scheduling chain ends.

The 30-minute maximum retry duration does not normally bind because the attempt count is exhausted
first.
A token endpoint that rejects requests quickly only needs to be unavailable for a few seconds at the refresh time.

### Reproduction

The standalone stub:

1. Issues an access token with `expires_in: 2`.
2. Allows one catalog listing.
3. Returns 503 from the token endpoint for eight seconds, covering the refresh attempts.
4. Restores the token endpoint.
5. Waits another eight seconds, covering four more token lifetimes.

Observed output on 1.10.0:

```text
token request 1 -> 200, expires in 2s
connected, token requests so far: 1
first listing ok
taking the token endpoint down for the refresh window
token request 2 -> 503
token request 3 -> 503
token request 4 -> 503
token request 5 -> 503
token request 6 -> 503
token request 7 -> 503
token request 8 -> 503
token endpoint back up after 8 token requests

token requests during the outage : 8
token requests after recovery : 0
RESULT: the refresh chain never restarted
```

The seven failed calls consist of six `Tasks` attempts and one credential-based fallback. After the
endpoint recovers, there are no further token requests.

### Expected behavior

A temporary token-endpoint failure should delay refresh rather than permanently disable it. Once the endpoint recovers, the session should obtain a new token without requiring the catalog object or host process to be restarted.

Permanent authentication failures should not cause aggressive retries forever. The retry behavior should distinguish a temporary endpoint failure from a session that has been closed or credentials that have been rejected.

### Production evidence

StarRocks has an independently reported production-facing failure with Iceberg Core 1.10.0 and the same root cause. After one refresh cycle fails, metadata requests eventually return 401 until the catalog is rebuilt or the process restarts:

https://github.com/StarRocks/starrocks/issues/76438

StarRocks also has its own missing catalog-recovery path, but that does not change the Iceberg client behavior: the Iceberg `AuthSession` stops refreshing after the failed cycle.

### Related Iceberg issues

- https://github.com/apache/iceberg/issues/12363 reports token expiry in Spark, but does not isolate the failed refresh scheduling chain.
- https://github.com/apache/iceberg/issues/13591 discusses whether expired-token recovery uses the correct OAuth flow.
- https://github.com/apache/iceberg/issues/12196 requests support for the OAuth `refresh_token` grant.

These are related OAuth discussions, but none adds recovery after the scheduled refresh chain stops.

### Fix direction for discussion

The scheduling code needs an outcome that distinguishes at least:

- Refresh succeeded, schedule from the new expiration.
- Refresh failed temporarily, schedule a bounded delayed retry.
- Refresh is disabled, or the session was closed, do not schedule again.

The current null return cannot make that distinction. It may represent a failed refresh, disabled refresh, a missing token, or a response without `expires_in`.

Simply rescheduling every time `refresh` returns null would therefore be unsafe. Token refresh uses the process-wide `ThreadPools.authRefreshPool()`, and closing an `AuthSession` changes its `keepRefreshed` state but does not shut down that shared executor. An unconditional retry could keep rescheduling a closed session.

A bounded backoff for temporary failures, with an explicit stopped outcome for closed sessions and permanent credential errors, would allow recovery without continuously calling an unavailable token
endpoint.

I would like maintainer guidance on the preferred outcome model and backoff policy before preparing a PR.

### Willingness to contribute

- [ ] I can contribute a fix for this bug independently
- [x] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time

Contributor guide

Open the contributing guide

Research direction

Start with core/src/main/java/org/apache/iceberg/rest/auth/OAuth2Util.java, then trace AuthSession.refresh, OAuth2Manager.contextualSession, and the Tasks.retry flow. Verify the behavior with the standalone HTTP-stub reproduction described in the issue; done means a temporary token-endpoint failure eventually recovers without refreshing closed sessions or retrying permanent authentication failures indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, authentication
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.