langgenius / langgenius/dify

Bug: OAuth token refresh credentials never persist to DB + concurrent refresh race condition

Open
#32,174 3 comments 2 reactions 2 assignees Claimed by @WH-2099 View on GitHub
🐞 bug project#dify
Dominant language
TypeScript
Stars
156k
Forks
24.6k
Avg merge
22h 9m
Merged PRs (30d)
610

Description

### Self Checks

- [x] This is only a bug report, not a feature request.
- [x] I have searched for [existing issues](https://github.com/langgenius/dify/issues) to avoid creating duplicates.
- [x] I understand this is an open source project and will provide as much detail as possible.

### Dify version

v1.12.1 (Docker deployment)

### Cloud or Self Hosted

Self Hosted (Docker)

### Steps to reproduce

1. Configure a built-in tool provider with OAuth2 credentials (e.g., QuickBooks)
2. Create a workflow with 2+ tool nodes using the same OAuth provider
3. Wait for the access token to expire (1 hour)
4. Execute the workflow
5. Check the database: `SELECT updated_at, created_at, expires_at FROM tool_builtin_providers WHERE provider LIKE '%quickbooks%';`
6. Observe `updated_at == created_at` — credentials were never updated despite successful refresh HTTP calls

### ✅ Expected Behavior

- After a successful OAuth token refresh, the new `encrypted_credentials` and `expires_at` should be persisted to the database via `db.session.commit()`
- Only ONE refresh should happen per token expiry cycle (~1 per hour)
- Concurrent requests detecting expired credentials should be serialized

### ❌ Actual Behavior

**Two critical bugs:**

#### Bug 1: `db.session.commit()` after refresh does not persist

The refresh code in `tool_manager.py` calls `db.session.commit()` after updating credentials, but the changes never persist to the database.

**Evidence from production database:**

```
| expires_at | updated_at | created_at |
|------------|---------------------|---------------------|
| 1770606647 | 2026-02-09 02:10:47 | 2026-02-09 02:10:47 |
```

- `created_at == updated_at`: the row has **never been updated** since creation
- `expires_at = created_at + 3600` (standard 1-hour token lifetime), never refreshed
- The `updated_at` column has `onupdate=func.current_timestamp()`, so any ORM commit would update it

**Yet 693 refresh HTTP calls were made in 24 hours** (528 from API container + 165 from Worker). All returned HTTP 200 OK from the plugin daemon. None persisted to the DB.

**Same-thread proof:** The same greenlet makes 4-6 sequential refresh calls ~1s apart. If the first commit worked, the second call would see fresh `expires_at` and skip:

```
10:37:17 [Dummy-1497] refresh_credentials "200 OK"
10:37:18 [Dummy-1497] refresh_credentials "200 OK" ← should have skipped
10:37:19 [Dummy-1497] refresh_credentials "200 OK" ← should have skipped
10:37:20 [Dummy-1497] refresh_credentials "200 OK" ← should have skipped
```

#### Bug 2: No locking on concurrent refresh

Multiple `GraphWorker` threads in workflow execution concurrently detect expired credentials and call `refresh_credentials` simultaneously:

```
02:18:24 [GraphWorker-0] refresh_credentials → SUCCESS (consumes old refresh_token)
02:18:25 [GraphWorker-0] tool/invoke → SUCCESS
02:18:28 [GraphWorker-1] refresh_credentials → FAIL "refresh token expired or revoked"
02:18:28 Node ABORT → workflow fails
```

Both workers read the same expired credentials from DB. GraphWorker-0 refreshes first (rotates the token). GraphWorker-1 retries with the now-invalidated old token → permanent failure.

This pattern was observed throughout the day with near-simultaneous refreshes from two workers:

```
05:30:44.680 [GraphWorker-0] refresh ← 3ms apart
05:30:44.683 [GraphWorker-1] refresh ← same req_id
```

These succeeded earlier due to the OAuth provider's grace period, but eventually (02:18), the stale token was rejected.

#### Cascading permanent failure

After the first failure, ALL subsequent requests fail because:
- The DB still stores the original (never-updated) refresh token from credential creation
- That token has been invalidated by the OAuth provider after 693+ uses
- Every new request reads the invalid token → fails → requires manual re-authorization

### Impact

- **Performance**: 693 unnecessary OAuth refresh calls/day (expected: ~24)
- **Reliability**: Eventual permanent failure requiring manual re-authorization
- **User experience**: All workflows using the affected OAuth provider stop working permanently

### Affected code

- `api/core/tools/tool_manager.py` — `get_tool_runtime()`, OAuth refresh block (lines ~270-298)
- `api/services/datasource_provider_service.py` — `get_datasource_credentials()`, same pattern

### Proposed fix

1. **Add Redis distributed lock** around the refresh block (non-blocking acquire + double-check pattern)
2. **Add exponential backoff polling** for requests that don't acquire the lock
3. **Investigate commit persistence** — ensure `db.session.commit()` persists in all execution contexts (API server, Celery worker, gevent greenlets)

A PR with the distributed lock fix is being prepared.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.