Bug: OAuth token refresh credentials never persist to DB + concurrent refresh race condition
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] This is only a bug report, not a feature request.
- [x] I have searched for [existing issues](https://github.com/langgenius/dify/issues) to avoid creating duplicates.
- [x] I understand this is an open source project and will provide as much detail as possible.
### Dify version
v1.12.1 (Docker deployment)
### Cloud or Self Hosted
Self Hosted (Docker)
### Steps to reproduce
1. Configure a built-in tool provider with OAuth2 credentials (e.g., QuickBooks)
2. Create a workflow with 2+ tool nodes using the same OAuth provider
3. Wait for the access token to expire (1 hour)
4. Execute the workflow
5. Check the database: `SELECT updated_at, created_at, expires_at FROM tool_builtin_providers WHERE provider LIKE '%quickbooks%';`
6. Observe `updated_at == created_at` — credentials were never updated despite successful refresh HTTP calls
### ✅ Expected Behavior
- After a successful OAuth token refresh, the new `encrypted_credentials` and `expires_at` should be persisted to the database via `db.session.commit()`
- Only ONE refresh should happen per token expiry cycle (~1 per hour)
- Concurrent requests detecting expired credentials should be serialized
### ❌ Actual Behavior
**Two critical bugs:**
#### Bug 1: `db.session.commit()` after refresh does not persist
The refresh code in `tool_manager.py` calls `db.session.commit()` after updating credentials, but the changes never persist to the database.
**Evidence from production database:**
```
| expires_at | updated_at | created_at |
|------------|---------------------|---------------------|
| 1770606647 | 2026-02-09 02:10:47 | 2026-02-09 02:10:47 |
```
- `created_at == updated_at`: the row has **never been updated** since creation
- `expires_at = created_at + 3600` (standard 1-hour token lifetime), never refreshed
- The `updated_at` column has `onupdate=func.current_timestamp()`, so any ORM commit would update it
**Yet 693 refresh HTTP calls were made in 24 hours** (528 from API container + 165 from Worker). All returned HTTP 200 OK from the plugin daemon. None persisted to the DB.
**Same-thread proof:** The same greenlet makes 4-6 sequential refresh calls ~1s apart. If the first commit worked, the second call would see fresh `expires_at` and skip:
```
10:37:17 [Dummy-1497] refresh_credentials "200 OK"
10:37:18 [Dummy-1497] refresh_credentials "200 OK" ← should have skipped
10:37:19 [Dummy-1497] refresh_credentials "200 OK" ← should have skipped
10:37:20 [Dummy-1497] refresh_credentials "200 OK" ← should have skipped
```
#### Bug 2: No locking on concurrent refresh
Multiple `GraphWorker` threads in workflow execution concurrently detect expired credentials and call `refresh_credentials` simultaneously:
```
02:18:24 [GraphWorker-0] refresh_credentials → SUCCESS (consumes old refresh_token)
02:18:25 [GraphWorker-0] tool/invoke → SUCCESS
02:18:28 [GraphWorker-1] refresh_credentials → FAIL "refresh token expired or revoked"
02:18:28 Node ABORT → workflow fails
```
Both workers read the same expired credentials from DB. GraphWorker-0 refreshes first (rotates the token). GraphWorker-1 retries with the now-invalidated old token → permanent failure.
This pattern was observed throughout the day with near-simultaneous refreshes from two workers:
```
05:30:44.680 [GraphWorker-0] refresh ← 3ms apart
05:30:44.683 [GraphWorker-1] refresh ← same req_id
```
These succeeded earlier due to the OAuth provider's grace period, but eventually (02:18), the stale token was rejected.
#### Cascading permanent failure
After the first failure, ALL subsequent requests fail because:
- The DB still stores the original (never-updated) refresh token from credential creation
- That token has been invalidated by the OAuth provider after 693+ uses
- Every new request reads the invalid token → fails → requires manual re-authorization
### Impact
- **Performance**: 693 unnecessary OAuth refresh calls/day (expected: ~24)
- **Reliability**: Eventual permanent failure requiring manual re-authorization
- **User experience**: All workflows using the affected OAuth provider stop working permanently
### Affected code
- `api/core/tools/tool_manager.py` — `get_tool_runtime()`, OAuth refresh block (lines ~270-298)
- `api/services/datasource_provider_service.py` — `get_datasource_credentials()`, same pattern
### Proposed fix
1. **Add Redis distributed lock** around the refresh block (non-blocking acquire + double-check pattern)
2. **Add exponential backoff polling** for requests that don't acquire the lock
3. **Investigate commit persistence** — ensure `db.session.commit()` persists in all execution contexts (API server, Celery worker, gevent greenlets)
A PR with the distributed lock fix is being prepared.
Contributor guide
Assessment
This issue has not been assessed yet.