kestra-io / kestra-io/plugin-airbyte

Re-attach: adopt an in-flight Airbyte sync instead of skipping or duplicating it

Open
#180 0 comments 0 reactions 1 assignee Claimed by @fdelbrayelle View on GitHub
area/plugin kind/customer-request
Dominant language
Java
Stars
3
Forks
7
Avg merge
8h 23m
Merged PRs (30d)
4

Description

### Describe the issue

### Problem

`connections.Sync` detects that a sync is already running for the connection, but cannot adopt it. In the catch block handling that case it returns `alreadyRunning: true` with `jobId: null` and no polling.

That leaves two outcomes, neither good:

- **Default** (`failOnActiveSync: true`): the task fails on a perfectly healthy in-flight sync.
- **Opt-out** (`failOnActiveSync: false`): the task returns success immediately while the real sync is still running — a false green that downstream freshness gates will trust, and no `jobId` for anything downstream to poll.

Detection is also brittle: it string-matches `"A sync is already running"` in the error message, so it breaks if Airbyte rewords it.

### Proposed

When a sync is already running for the connection, adopt it: capture its job id from the jobs-list API and poll it to completion exactly as if this task had started it, flagging adoption in the output. `connections.CheckStatus` already polls a job id, so the polling half exists.

1. Look up the connection's running job.
2. If found, adopt and poll it.
3. Otherwise trigger a new sync and poll that.

Using the jobs-list API for detection also removes the error-string matching.

Match on the connection's running job rather than on this execution's taskrun id. Worker-loss resubmit preserves the taskrun id, so own-run keying would cover that case; external matching additionally covers retries with `behavior: CREATE_NEW_EXECUTION`, replays, two executions driving the same connection, and syncs started outside Kestra.

Use `ResumableTaskInterface` for the restart-survival half once https://github.com/kestra-io/kestra/pull/17995 lands; the adopt-an-external-run policy sits on top and stays in the task.

### To decide in review

- Should adopting a sync this execution did not start be default or opt-in?
- What should `kill` do to an adopted sync?
- Is an already-succeeded job adoptable, or only one still running?

### Related

- https://github.com/kestra-io/kestra/pull/17995 — the shared contract
- https://github.com/kestra-io/plugin-fivetran/issues/99 — the Fivetran equivalent
- `bigquery/RunTransferConfig` in https://github.com/kestra-io/plugin-gcp — the external-matching precedent

### Environment

- Kestra Version: develop

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.