airbytehq / airbytehq/airbyte

Discover Behavior Change with DISCOVER_REFRESH_WINDOW_MINUTES=0 in v1.8.0

Ouverte
#72,306 3 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
autoteam community needs-triage team/use
Langage dominant
Python
Étoiles
22.1k
Forks
5.3k
Métriques de merge des PR
Métriques de PR en attente

Description

### Topic

Discover runs once even if the DISCOVER_REFRESH_WINDOW_MINUTES variable is set = 0

### Relevant information

### Problem Description

In version **v1.8.0**, the behavior of the `DISCOVER_REFRESH_WINDOW_MINUTES=0` configuration changed significantly, causing unexpected discover execution on the first sync after upgrade.

### Expected Behavior (V1 - before v1.8.0)

With `DISCOVER_REFRESH_WINDOW_MINUTES=0`, the `DiscoverCommand` (V1) implemented a **true NO-OP**:

```kotlin
// airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommand.kt
override fun start(input: DiscoverCatalogInput, signalPayload: String?): String {
if (input.launcherConfig.isReset() ||
(isAutoRefresh(input) && discoverAutoRefreshWindow == Duration.INFINITE)) {
return NOOP_DISCOVER_PLACEHOLDER_ID // ← Returns immediately without execution
}
return super.start(input, signalPayload)
}
```

**Result:**
- ❌ **Never executes** the discover
- ✅ Returns placeholder instantly
- ✅ **Zero resource** consumption
- ✅ **Zero calls** to the source connector

### Current Behavior (V2 - from v1.8.0)

In v1.8.0, `SyncWorkflowV2` now uses `DiscoverCommandV2` which **does not have** the NO-OP logic:

```kotlin
// airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommandV2.kt
override fun start(input: DiscoverSourceApiInput, signalPayload: String?): String {
val commandId = "discover_${input.jobId}_${input.attemptId}_${input.actorId}"

airbyteApiClient.commandApi.runDiscoverCommand(...) // ← Always executes the call

return commandId
}
```

What happens:
1. **First sync**: Executes discover and creates command in database
2. **Subsequent syncs**: Reuse the result due to `commandsRepository.existsById(commandId)` returning `true`

**Result:**
- ✅ **Executes discover ON FIRST SYNC** (unexpected!)
- ⚠️ Creates workload and consumes resources once
- ⚠️ Makes connector call on first sync
- ✅ Subsequent syncs reuse the result (infinite cache)
- ❌ **NOT a true NO-OP**

### Visual Comparison

| Aspect | V1 (DiscoverCommand) | V2 (DiscoverCommandV2) |
|---------|---------------------|------------------------|
| **First sync** | ❌ Does not execute | ✅ **Executes** |
| **Following syncs** | ❌ Does not execute | ❌ Does not execute (reuses) |
| **Is true NO-OP?** | ✅ Yes | ❌ No - executes once |
| **Behavior** | Complete skip | Infinite cache |
| **Resource consumption** | Zero | Once |

### Impact

1. **Undocumented Breaking Change**: Users who configured `DISCOVER_REFRESH_WINDOW_MINUTES=0` to completely disable discover will have discover executed on the first sync after upgrading to v1.8.0

2. **Unexpected Costs**: For large sources or those with many tables, discover can be costly in time and resources

3. **Inconsistent Behavior**: The configuration variable exists but doesn't have the same effect

### Proposed Solution

Restore the NO-OP logic in `DiscoverCommandV2` or `CommandService` when `DISCOVER_REFRESH_WINDOW_MINUTES=0`:

### Steps to Reproduce

1. Configure `DISCOVER_REFRESH_WINDOW_MINUTES=0` in environment
2. Upgrade Airbyte to v1.8.0
3. Run a sync on any connection
4. Observe that discover is executed on the first sync (can be seen in logs)
5. Run another sync on the same connection
6. Observe that discover is not executed (expected behavior)

### Environment

- **Airbyte Version**: v1.8.0
- **Deployment**: Self-hosted
- **Configuration**: `DISCOVER_REFRESH_WINDOW_MINUTES=0`

### Relevant Files

- `airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommand.kt` (V1 - old)
- `airbyte-workers/src/main/kotlin/io/airbyte/workers/commands/DiscoverCommandV2.kt` (V2 - new)
- `airbyte-workers/src/main/kotlin/io/airbyte/workers/temporal/sync/SyncWorkflowV2Impl.kt`
- `airbyte-server/src/main/kotlin/io/airbyte/server/services/CommandService.kt`
- `airbyte-server/src/main/kotlin/io/airbyte/server/helpers/WorkloadIdGenerator.kt`

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/11017

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.