Discover Behavior Change with DISCOVER_REFRESH_WINDOW_MINUTES=0 in v1.8.0
- Langage dominant
- Python
- Étoiles
- 22.1k
- Forks
- 5.3k
- Métriques de merge des PR
- Métriques de PR en attente
Description
### Topic
Discover runs once even if the DISCOVER_REFRESH_WINDOW_MINUTES variable is set = 0
### Relevant information
### Problem Description
In version **v1.8.0**, the behavior of the `DISCOVER_REFRESH_WINDOW_MINUTES=0` configuration changed significantly, causing unexpected discover execution on the first sync after upgrade.
### Expected Behavior (V1 - before v1.8.0)
With `DISCOVER_REFRESH_WINDOW_MINUTES=0`, the `DiscoverCommand` (V1) implemented a **true NO-OP**:
```kotlin
// airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommand.kt
override fun start(input: DiscoverCatalogInput, signalPayload: String?): String {
if (input.launcherConfig.isReset() ||
(isAutoRefresh(input) && discoverAutoRefreshWindow == Duration.INFINITE)) {
return NOOP_DISCOVER_PLACEHOLDER_ID // ← Returns immediately without execution
}
return super.start(input, signalPayload)
}
```
**Result:**
- ❌ **Never executes** the discover
- ✅ Returns placeholder instantly
- ✅ **Zero resource** consumption
- ✅ **Zero calls** to the source connector
### Current Behavior (V2 - from v1.8.0)
In v1.8.0, `SyncWorkflowV2` now uses `DiscoverCommandV2` which **does not have** the NO-OP logic:
```kotlin
// airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommandV2.kt
override fun start(input: DiscoverSourceApiInput, signalPayload: String?): String {
val commandId = "discover_${input.jobId}_${input.attemptId}_${input.actorId}"
airbyteApiClient.commandApi.runDiscoverCommand(...) // ← Always executes the call
return commandId
}
```
What happens:
1. **First sync**: Executes discover and creates command in database
2. **Subsequent syncs**: Reuse the result due to `commandsRepository.existsById(commandId)` returning `true`
**Result:**
- ✅ **Executes discover ON FIRST SYNC** (unexpected!)
- ⚠️ Creates workload and consumes resources once
- ⚠️ Makes connector call on first sync
- ✅ Subsequent syncs reuse the result (infinite cache)
- ❌ **NOT a true NO-OP**
### Visual Comparison
| Aspect | V1 (DiscoverCommand) | V2 (DiscoverCommandV2) |
|---------|---------------------|------------------------|
| **First sync** | ❌ Does not execute | ✅ **Executes** |
| **Following syncs** | ❌ Does not execute | ❌ Does not execute (reuses) |
| **Is true NO-OP?** | ✅ Yes | ❌ No - executes once |
| **Behavior** | Complete skip | Infinite cache |
| **Resource consumption** | Zero | Once |
### Impact
1. **Undocumented Breaking Change**: Users who configured `DISCOVER_REFRESH_WINDOW_MINUTES=0` to completely disable discover will have discover executed on the first sync after upgrading to v1.8.0
2. **Unexpected Costs**: For large sources or those with many tables, discover can be costly in time and resources
3. **Inconsistent Behavior**: The configuration variable exists but doesn't have the same effect
### Proposed Solution
Restore the NO-OP logic in `DiscoverCommandV2` or `CommandService` when `DISCOVER_REFRESH_WINDOW_MINUTES=0`:
### Steps to Reproduce
1. Configure `DISCOVER_REFRESH_WINDOW_MINUTES=0` in environment
2. Upgrade Airbyte to v1.8.0
3. Run a sync on any connection
4. Observe that discover is executed on the first sync (can be seen in logs)
5. Run another sync on the same connection
6. Observe that discover is not executed (expected behavior)
### Environment
- **Airbyte Version**: v1.8.0
- **Deployment**: Self-hosted
- **Configuration**: `DISCOVER_REFRESH_WINDOW_MINUTES=0`
### Relevant Files
- `airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommand.kt` (V1 - old)
- `airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommandV2.kt` (V2 - new)
- `airbyte-workers/src/main/kotlin/io/airbyte/workers/temporal/sync/SyncWorkflowV2Impl.kt`
- `airbyte-server/src/main/kotlin/io/airbyte/server/services/CommandService.kt`
- `airbyte-server/src/main/kotlin/io/airbyte/server/helpers/WorkloadIdGenerator.kt`
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11017
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.