pipeline: UpdateStatus mutates in-memory status before the store write and never rolls it back
- Dominant language
- Go
- Stars
- 610
- Forks
- 63
- Avg merge
- 12h 28m
- Merged PRs (30d)
- 57
Description
Found while planning the fix for #2806, verified against `main`.
`pipeline.Service.UpdateStatus` (`pkg/pipeline/service.go:366-382`):
```go
pipeline, err := s.Get(ctx, id) // shared *Instance pointer, s.instances[id] — no copy
...
pipeline.SetStatus(status) // in-memory mutation, FIRST
pipeline.Error = errMsg
...
err = s.store.Set(ctx, pipeline.ID, pipeline) // durable write, SECOND
if err != nil {
return cerrors.Errorf("pipeline not updated: %w", err) // no rollback of SetStatus
}
```
If the store write fails, the in-memory instance keeps the new status while the persisted record keeps the old one. Two consequences observed while working on #2806:
1. **A failed `UpdateStatus(StatusRunning)` leaves the in-memory instance reporting Running.** `lifecycle.Service.Start`'s precondition (`pkg/lifecycle/service.go:194`, `if pl.GetStatus() == pipeline.StatusRunning`) reads that in-memory status, so a retry of `Start` is rejected with `ErrPipelineRunning` for a pipeline that never actually started. The operator's recovery path from a transient store failure is blocked by the failure itself.
2. **`Get` hands out the shared pointer** (`service.go:106-119` returns `s.instances[id]` directly), so every holder of that `*Instance` — including a previous run's `runnablePipeline` — observes the mutation immediately, whether or not it was ever persisted.
Not filed as part of #2806 because it is a distinct bug with a distinct fix (write-then-mutate, or mutate-and-roll-back-on-error), and folding it into a lifecycle change would make both harder to review.
Tier 1: pipeline status is what operators and the provisioning path act on, and a torn write here makes the in-memory and persisted views disagree silently.
Contributor guide
Research direction
Start in pkg/pipeline/service.go:366-382 and inspect Get at lines 106-119, then compare the lifecycle precondition in pkg/lifecycle/service.go:194. Reproduce a failed store.Set during UpdateStatus and verify that the in-memory and persisted statuses remain consistent, including that a later Start retry is not blocked by an unsuccessful update.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100