ConnectionManagerWorkflow continues retrying after deletion when JobFailure times out
- Vorherrschende Sprache
- Python
- Sterne
- 22.1k
- Forks
- 5.3k
- PR-Merge-Kennzahlen
- PR-Kennzahlen ausstehend
Beschreibung
## Bug description
A deleted connection can retain an active `ConnectionManagerWorkflow` that continues restarting its retry path indefinitely.
The workflow successfully receives the `deleteConnection` signal and logs that the connection is marked deleted and its job is being cancelled. However, if the `JobFailure` activity then encounters Temporal heartbeat timeouts, the outer workflow handles that activity failure by waiting and restarting. It later starts another attempt for the already-deleted connection.
Observed with Airbyte `2.1.0`.
## Expected behavior
After `deleteConnection` is received:
- Pending activities and retry timers are cancelled.
- The connection-manager workflow completes or terminates.
- No subsequent retry attempt is scheduled for the deleted connection.
## Actual behavior
- `deleteConnection` is processed.
- `JobFailure` repeatedly reaches an activity heartbeat timeout.
- The workflow waits 10 minutes and restarts.
- A new failure/retry activity is scheduled despite the deleted state.
- Retry counters can grow far beyond their configured total-failure limit.
- The workflow history grows continuously.
## Sanitized log excerpts
```text
INFO [signal deleteConnection]
Set as deleted and canceling job for connection
ERROR [ConnectionManagerWorkflow]
Activity `JobFailure` failed: Activity task timed out
Caused by: activity Heartbeat timeout
INFO [ConnectionManagerWorkflow]
Waiting PT10M before restarting the workflow for connection
INFO [activity executor]
Retry state shows failures exceeding the configured total-failure limit
INFO [activity executor]
A new AttemptFailureWithAttemptNumber activity is started after deletion
```
## Observed Temporal state
- Workflow type: `ConnectionManagerWorkflow`
- Task queue: `CONNECTION_UPDATER`
- The deletion signal appears in the workflow history.
- After the signal, the workflow records a `JobFailure` heartbeat timeout, starts a restart-delay timer, then schedules another failure/retry activity.
- The workflow remains running rather than completing.
## Suggested fix
Ensure the deletion state is checked before scheduling or restarting any retry/failure activity. A deletion signal should cancel the outer workflow retry path even if a concurrent `JobFailure` activity has timed out.
No connection IDs, workflow IDs, workspace identifiers, hostnames, IP addresses, timestamps, or logs containing company-specific data are included.
---
**Internal Tracking:** https://github.com/airbytehq/oncall/issues/13228
Beitragsleitfaden
Bewertung
Dieses Issue wurde noch nicht bewertet.