kestra-io / kestra-io/plugin-fivetran

Re-attach: adopt an in-flight Fivetran sync instead of triggering a duplicate

Open
#99 0 comments 0 reactions 1 assignee Claimed by @fdelbrayelle View on GitHub
area/plugin kind/customer-request
Dominant language
Java
Stars
1
Forks
5
Avg merge
1d 11h
Merged PRs (30d)
5

Description

### Problem

`connectors.Sync` has no deliberate re-attach. What happens after a worker loss depends on timing and on `force`:

- If the original sync is **still running** and `force` is false (the default), Fivetran skips the POST server-side, and the task then polls the connector's `completedDate` — so it accidentally waits on the original sync. Right outcome, by accident, with no output saying so.
- If the original sync **completed in the gap** between worker loss and resubmit, a second sync is triggered.
- With `force: true`, the in-progress sync is restarted.

`force` is not re-attach: it restarts an in-progress sync, which is the opposite of adopting one.

### Today

`Sync.java` posts to the sync endpoint with `force` as the only in-flight-related property. There is no lookup of a running sync, and nothing in the task distinguishes "I started this sync" from "I am waiting on someone else's".

### Proposed

Adopt-first ordering, made explicit rather than incidental:

1. Read the connector's sync state (GET connector).
2. If a sync is in progress, adopt it and poll to completion, flagging adoption in the task output.
3. Otherwise start a new sync and poll that.

Match on the connector's running sync rather than on this execution's taskrun id. Worker-loss resubmit preserves the taskrun id, so own-run keying would cover that case; external matching additionally covers retries with `behavior: CREATE_NEW_EXECUTION`, replays and manual re-runs, two executions driving the same connector, and syncs Fivetran started on its own schedule.

`bigquery/RunTransferConfig` in plugin-gcp is the precedent for external matching, including its `reattachMaxAge` staleness guard. Note the three existing implementations (dbt, gcp, hex) are three different shapes, so this should pick one deliberately rather than "match the others".

Use `ResumableTaskInterface` for the restart-survival half once https://github.com/kestra-io/kestra/pull/17995 lands; the adopt-an-external-run policy sits on top and stays in the task.

### To decide in review

- Is an already-succeeded sync adoptable? plugin-dbt says yes, plugin-gcp says no.
- What should `kill` do to an adopted sync this execution did not start? Cancelling someone else's sync is worse than leaving it running.
- Default or opt-in? plugin-gcp ships adopt-any as default true; adopting a run you did not start is safer against duplicates but surprising when two flows legitimately drive the same connector.

### Related

- https://github.com/kestra-io/kestra/pull/17995 — the shared contract
- https://github.com/kestra-io/plugin-dbt/issues/307 — the dbt equivalent (closed via plugin-dbt#310; confirm the release tag before citing a version)
- https://github.com/kestra-io/plugin-airbyte/issues/180 — the Airbyte equivalent
- https://github.com/kestra-io/kestra-ee/issues/9922 — the customer Epic

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.