kestra-io / kestra-io/plugin-airbyte
Re-attach: adopt an in-flight Airbyte sync instead of skipping or duplicating it
- Dominant language
- Java
- Stars
- 3
- Forks
- 7
- Avg merge
- 8h 23m
- Merged PRs (30d)
- 4
Description
### Describe the issue
### Problem
`connections.Sync` detects that a sync is already running for the connection, but cannot adopt it. In the catch block handling that case it returns `alreadyRunning: true` with `jobId: null` and no polling.
That leaves two outcomes, neither good:
- **Default** (`failOnActiveSync: true`): the task fails on a perfectly healthy in-flight sync.
- **Opt-out** (`failOnActiveSync: false`): the task returns success immediately while the real sync is still running — a false green that downstream freshness gates will trust, and no `jobId` for anything downstream to poll.
Detection is also brittle: it string-matches `"A sync is already running"` in the error message, so it breaks if Airbyte rewords it.
### Proposed
When a sync is already running for the connection, adopt it: capture its job id from the jobs-list API and poll it to completion exactly as if this task had started it, flagging adoption in the output. `connections.CheckStatus` already polls a job id, so the polling half exists.
1. Look up the connection's running job.
2. If found, adopt and poll it.
3. Otherwise trigger a new sync and poll that.
Using the jobs-list API for detection also removes the error-string matching.
Match on the connection's running job rather than on this execution's taskrun id. Worker-loss resubmit preserves the taskrun id, so own-run keying would cover that case; external matching additionally covers retries with `behavior: CREATE_NEW_EXECUTION`, replays, two executions driving the same connection, and syncs started outside Kestra.
Use `ResumableTaskInterface` for the restart-survival half once https://github.com/kestra-io/kestra/pull/17995 lands; the adopt-an-external-run policy sits on top and stays in the task.
### To decide in review
- Should adopting a sync this execution did not start be default or opt-in?
- What should `kill` do to an adopted sync?
- Is an already-succeeded job adoptable, or only one still running?
### Related
- https://github.com/kestra-io/kestra/pull/17995 — the shared contract
- https://github.com/kestra-io/plugin-fivetran/issues/99 — the Fivetran equivalent
- `bigquery/RunTransferConfig` in https://github.com/kestra-io/plugin-gcp — the external-matching precedent
### Environment
- Kestra Version: develop
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.