Discover Behavior Change with DISCOVER_REFRESH_WINDOW_MINUTES=0 in v1.8.0
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.4k
- Merge médio
- 5h
- PRs com merge (30d)
- 671
Descrição
### Topic
Discover runs once even if the DISCOVER_REFRESH_WINDOW_MINUTES variable is set = 0
### Relevant information
### Problem Description
In version **v1.8.0**, the behavior of the `DISCOVER_REFRESH_WINDOW_MINUTES=0` configuration changed significantly, causing unexpected discover execution on the first sync after upgrade.
### Expected Behavior (V1 - before v1.8.0)
With `DISCOVER_REFRESH_WINDOW_MINUTES=0`, the `DiscoverCommand` (V1) implemented a **true NO-OP**:
```kotlin
// airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommand.kt
override fun start(input: DiscoverCatalogInput, signalPayload: String?): String {
if (input.launcherConfig.isReset() ||
(isAutoRefresh(input) && discoverAutoRefreshWindow == Duration.INFINITE)) {
return NOOP_DISCOVER_PLACEHOLDER_ID // ← Returns immediately without execution
}
return super.start(input, signalPayload)
}
```
**Result:**
- ❌ **Never executes** the discover
- ✅ Returns placeholder instantly
- ✅ **Zero resource** consumption
- ✅ **Zero calls** to the source connector
### Current Behavior (V2 - from v1.8.0)
In v1.8.0, `SyncWorkflowV2` now uses `DiscoverCommandV2` which **does not have** the NO-OP logic:
```kotlin
// airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommandV2.kt
override fun start(input: DiscoverSourceApiInput, signalPayload: String?): String {
val commandId = "discover_${input.jobId}_${input.attemptId}_${input.actorId}"
airbyteApiClient.commandApi.runDiscoverCommand(...) // ← Always executes the call
return commandId
}
```
What happens:
1. **First sync**: Executes discover and creates command in database
2. **Subsequent syncs**: Reuse the result due to `commandsRepository.existsById(commandId)` returning `true`
**Result:**
- ✅ **Executes discover ON FIRST SYNC** (unexpected!)
- ⚠️ Creates workload and consumes resources once
- ⚠️ Makes connector call on first sync
- ✅ Subsequent syncs reuse the result (infinite cache)
- ❌ **NOT a true NO-OP**
### Visual Comparison
| Aspect | V1 (DiscoverCommand) | V2 (DiscoverCommandV2) |
|---------|---------------------|------------------------|
| **First sync** | ❌ Does not execute | ✅ **Executes** |
| **Following syncs** | ❌ Does not execute | ❌ Does not execute (reuses) |
| **Is true NO-OP?** | ✅ Yes | ❌ No - executes once |
| **Behavior** | Complete skip | Infinite cache |
| **Resource consumption** | Zero | Once |
### Impact
1. **Undocumented Breaking Change**: Users who configured `DISCOVER_REFRESH_WINDOW_MINUTES=0` to completely disable discover will have discover executed on the first sync after upgrading to v1.8.0
2. **Unexpected Costs**: For large sources or those with many tables, discover can be costly in time and resources
3. **Inconsistent Behavior**: The configuration variable exists but doesn't have the same effect
### Proposed Solution
Restore the NO-OP logic in `DiscoverCommandV2` or `CommandService` when `DISCOVER_REFRESH_WINDOW_MINUTES=0`:
### Steps to Reproduce
1. Configure `DISCOVER_REFRESH_WINDOW_MINUTES=0` in environment
2. Upgrade Airbyte to v1.8.0
3. Run a sync on any connection
4. Observe that discover is executed on the first sync (can be seen in logs)
5. Run another sync on the same connection
6. Observe that discover is not executed (expected behavior)
### Environment
- **Airbyte Version**: v1.8.0
- **Deployment**: Self-hosted
- **Configuration**: `DISCOVER_REFRESH_WINDOW_MINUTES=0`
### Relevant Files
- `airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommand.kt` (V1 - old)
- `airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommandV2.kt` (V2 - new)
- `airbyte-workers/src/main/kotlin/io/airbyte/workers/temporal/sync/SyncWorkflowV2Impl.kt`
- `airbyte-server/src/main/kotlin/io/airbyte/server/services/CommandService.kt`
- `airbyte-server/src/main/kotlin/io/airbyte/server/helpers/WorkloadIdGenerator.kt`
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11017
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.