lifecycle-poc (arch-v2): cleanup still deletes runningPipelines by key, so a superseded run can erase a live one
- Dominant language
- Go
- Stars
- 610
- Forks
- 63
- Avg merge
- 12h 28m
- Merged PRs (30d)
- 57
Description
Found while fixing the v1 instance (#2806). **v2 has the delete-side half of the same bug**, even though #2807 already fixed its publication ordering.
## The code
`pkg/lifecycle-poc/service.go:1658` — the recovery-failed arm falls through to an unconditional:
```go
s.runningPipelines.Delete(rp.pipeline.ID)
```
and v2's `Set` at `:1702` has no rollback by design — its comment at `:1697` explicitly delegates cleanup to the goroutine "including when the UpdateStatus below fails".
## The scenario
Identical to the one that shaped the v1 fix: in a recovery chain, the outer cleanup goroutine runs `recoverPipeline -> StartWithBackoff -> Start(rp2)` **synchronously on its own goroutine**. If the nested run's `UpdateStatus` fails, the error unwinds back into the outer cleanup, which falls through to its terminal block and deletes **by key** — erasing `rp2`, whose nodes are alive.
Result: `rp2` is unreachable via `Stop`/`WaitPipeline`, and `WaitPipeline` returns a false nil off `terminalErrors`.
## The fix
Port v1's `deleteRunningPipelineIfCurrent` plus its `publishMu`: compare-and-delete, serialized against publication so it is a real CAS rather than a TOCTOU. `csync.Map` has no CAS primitive, which is why the mutex is needed rather than just the comparison.
## Note on the plans
The v0.20 plan's AC 6 for #2806 said the two packages would end up sharing one rule. They now share **publication ordering** but have **diverged on delete discipline** — v1 compares, v2 does not. That divergence should not outlive this issue.
Tier 1. Related: #2806, #2807.
Contributor guide
Research direction
Start in pkg/lifecycle-poc/service.go around lines 1658-1702 and trace recoverPipeline, StartWithBackoff, Start, Set, Stop, and WaitPipeline. Compare the v2 cleanup with v1's fix from #2806, including publication synchronization. Done means recovery cleanup cannot delete a newer running pipeline by key, and live runs remain reachable through Stop and WaitPipeline.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100