airbytehq / airbytehq/airbyte

ConnectionManagerWorkflow continues retrying after deletion when JobFailure times out

Offen
#83,340 3 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
autoteam community team/use
Vorherrschende Sprache
Python
Sterne
22.1k
Forks
5.3k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

## Bug description

A deleted connection can retain an active `ConnectionManagerWorkflow` that continues restarting its retry path indefinitely.

The workflow successfully receives the `deleteConnection` signal and logs that the connection is marked deleted and its job is being cancelled. However, if the `JobFailure` activity then encounters Temporal heartbeat timeouts, the outer workflow handles that activity failure by waiting and restarting. It later starts another attempt for the already-deleted connection.

Observed with Airbyte `2.1.0`.

## Expected behavior

After `deleteConnection` is received:

- Pending activities and retry timers are cancelled.
- The connection-manager workflow completes or terminates.
- No subsequent retry attempt is scheduled for the deleted connection.

## Actual behavior

- `deleteConnection` is processed.
- `JobFailure` repeatedly reaches an activity heartbeat timeout.
- The workflow waits 10 minutes and restarts.
- A new failure/retry activity is scheduled despite the deleted state.
- Retry counters can grow far beyond their configured total-failure limit.
- The workflow history grows continuously.

## Sanitized log excerpts

```text
INFO [signal deleteConnection]
Set as deleted and canceling job for connection

ERROR [ConnectionManagerWorkflow]
Activity `JobFailure` failed: Activity task timed out
Caused by: activity Heartbeat timeout

INFO [ConnectionManagerWorkflow]
Waiting PT10M before restarting the workflow for connection

INFO [activity executor]
Retry state shows failures exceeding the configured total-failure limit

INFO [activity executor]
A new AttemptFailureWithAttemptNumber activity is started after deletion
```

## Observed Temporal state

- Workflow type: `ConnectionManagerWorkflow`
- Task queue: `CONNECTION_UPDATER`
- The deletion signal appears in the workflow history.
- After the signal, the workflow records a `JobFailure` heartbeat timeout, starts a restart-delay timer, then schedules another failure/retry activity.
- The workflow remains running rather than completing.

## Suggested fix

Ensure the deletion state is checked before scheduling or restarting any retry/failure activity. A deletion signal should cancel the outer workflow retry path even if a concurrent `JobFailure` activity has timed out.

No connection IDs, workflow IDs, workspace identifiers, hostnames, IP addresses, timestamps, or logs containing company-specific data are included.

---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13228

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.