ConduitIO / ConduitIO/conduit

pipeline: UpdateStatus mutates in-memory status before the store write and never rolls it back

Open
#2,809 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

Found while planning the fix for #2806, verified against `main`.

`pipeline.Service.UpdateStatus` (`pkg/pipeline/service.go:366-382`):

```go
pipeline, err := s.Get(ctx, id) // shared *Instance pointer, s.instances[id] — no copy
...
pipeline.SetStatus(status) // in-memory mutation, FIRST
pipeline.Error = errMsg
...
err = s.store.Set(ctx, pipeline.ID, pipeline) // durable write, SECOND
if err != nil {
return cerrors.Errorf("pipeline not updated: %w", err) // no rollback of SetStatus
}
```

If the store write fails, the in-memory instance keeps the new status while the persisted record keeps the old one. Two consequences observed while working on #2806:

1. **A failed `UpdateStatus(StatusRunning)` leaves the in-memory instance reporting Running.** `lifecycle.Service.Start`'s precondition (`pkg/lifecycle/service.go:194`, `if pl.GetStatus() == pipeline.StatusRunning`) reads that in-memory status, so a retry of `Start` is rejected with `ErrPipelineRunning` for a pipeline that never actually started. The operator's recovery path from a transient store failure is blocked by the failure itself.
2. **`Get` hands out the shared pointer** (`service.go:106-119` returns `s.instances[id]` directly), so every holder of that `*Instance` — including a previous run's `runnablePipeline` — observes the mutation immediately, whether or not it was ever persisted.

Not filed as part of #2806 because it is a distinct bug with a distinct fix (write-then-mutate, or mutate-and-roll-back-on-error), and folding it into a lifecycle change would make both harder to review.

Tier 1: pipeline status is what operators and the provisioning path act on, and a torn write here makes the in-memory and persisted views disagree silently.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/pipeline/service.go:366-382 and inspect Get at lines 106-119, then compare the lifecycle precondition in pkg/lifecycle/service.go:194. Reproduce a failed store.Set during UpdateStatus and verify that the in-memory and persisted statuses remain consistent, including that a later Start retry is not blocked by an unsuccessful update.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.