ConduitIO / ConduitIO/conduit

processor/connector Service: in-memory instance cache can diverge from store on txn.Commit failure

Open
#2,618 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

Pre-existing, surfaced by the #2617 review. `processor.Service.Get` returns the cached `*Instance` and `updateConfig` mutates `instance.Plugin`/`Config` in place BEFORE `store.Set`. The in-memory mutation is not part of provisioning's DB transaction, and `importPipeline`'s reverse-rollback fires only on `executeActions` failure, not on `txn.Commit()` failure. So if `executeActions` succeeds but `Commit` fails (rare: disk full / closed DB), `txn.Discard()` reverts the store while the cache keeps the new config → cache/store divergence that self-heals only on process restart (Init reloads from store).

Byte-for-byte identical to the pre-existing `processor.Service.Update` and connector `Update`, so #2617 didn't introduce it — but by removing the running-guard for the live-swap path it newly lets a *running* processor reach it. Config-plane only (not record/ack/position/checkpoint integrity).

Fix options: revert the in-memory mutation on commit failure, or re-`Get` from store after commit. Applies to processor AND connector services.

Contributor guide

Open the contributing guide

Research direction

Inspect processor.Service.Get, updateConfig, Update, and connector Update, then trace importPipeline through executeActions, txn.Commit, and txn.Discard. Reproduce the commit-failure path and verify that processor and connector in-memory instances match the store after a failed transaction, including coverage for both services.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.