kestra-io / kestra-io/plugin-fivetran
Re-attach: adopt an in-flight Fivetran sync instead of triggering a duplicate
- Dominant language
- Java
- Stars
- 1
- Forks
- 5
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 5
Description
### Problem
`connectors.Sync` has no deliberate re-attach. What happens after a worker loss depends on timing and on `force`:
- If the original sync is **still running** and `force` is false (the default), Fivetran skips the POST server-side, and the task then polls the connector's `completedDate` — so it accidentally waits on the original sync. Right outcome, by accident, with no output saying so.
- If the original sync **completed in the gap** between worker loss and resubmit, a second sync is triggered.
- With `force: true`, the in-progress sync is restarted.
`force` is not re-attach: it restarts an in-progress sync, which is the opposite of adopting one.
### Today
`Sync.java` posts to the sync endpoint with `force` as the only in-flight-related property. There is no lookup of a running sync, and nothing in the task distinguishes "I started this sync" from "I am waiting on someone else's".
### Proposed
Adopt-first ordering, made explicit rather than incidental:
1. Read the connector's sync state (GET connector).
2. If a sync is in progress, adopt it and poll to completion, flagging adoption in the task output.
3. Otherwise start a new sync and poll that.
Match on the connector's running sync rather than on this execution's taskrun id. Worker-loss resubmit preserves the taskrun id, so own-run keying would cover that case; external matching additionally covers retries with `behavior: CREATE_NEW_EXECUTION`, replays and manual re-runs, two executions driving the same connector, and syncs Fivetran started on its own schedule.
`bigquery/RunTransferConfig` in plugin-gcp is the precedent for external matching, including its `reattachMaxAge` staleness guard. Note the three existing implementations (dbt, gcp, hex) are three different shapes, so this should pick one deliberately rather than "match the others".
Use `ResumableTaskInterface` for the restart-survival half once https://github.com/kestra-io/kestra/pull/17995 lands; the adopt-an-external-run policy sits on top and stays in the task.
### To decide in review
- Is an already-succeeded sync adoptable? plugin-dbt says yes, plugin-gcp says no.
- What should `kill` do to an adopted sync this execution did not start? Cancelling someone else's sync is worse than leaving it running.
- Default or opt-in? plugin-gcp ships adopt-any as default true; adopting a run you did not start is safer against duplicates but surprising when two flows legitimately drive the same connector.
### Related
- https://github.com/kestra-io/kestra/pull/17995 — the shared contract
- https://github.com/kestra-io/plugin-dbt/issues/307 — the dbt equivalent (closed via plugin-dbt#310; confirm the release tag before citing a version)
- https://github.com/kestra-io/plugin-airbyte/issues/180 — the Airbyte equivalent
- https://github.com/kestra-io/kestra-ee/issues/9922 — the customer Epic
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.