ConduitIO / ConduitIO/conduit

lifecycle-poc (arch-v2): cleanup still deletes runningPipelines by key, so a superseded run can erase a live one

Open
#2,811 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
610
Forks
63
Avg merge
12h 28m
Merged PRs (30d)
57

Description

Found while fixing the v1 instance (#2806). **v2 has the delete-side half of the same bug**, even though #2807 already fixed its publication ordering.

## The code

`pkg/lifecycle-poc/service.go:1658` — the recovery-failed arm falls through to an unconditional:

```go
s.runningPipelines.Delete(rp.pipeline.ID)
```

and v2's `Set` at `:1702` has no rollback by design — its comment at `:1697` explicitly delegates cleanup to the goroutine "including when the UpdateStatus below fails".

## The scenario

Identical to the one that shaped the v1 fix: in a recovery chain, the outer cleanup goroutine runs `recoverPipeline -> StartWithBackoff -> Start(rp2)` **synchronously on its own goroutine**. If the nested run's `UpdateStatus` fails, the error unwinds back into the outer cleanup, which falls through to its terminal block and deletes **by key** — erasing `rp2`, whose nodes are alive.

Result: `rp2` is unreachable via `Stop`/`WaitPipeline`, and `WaitPipeline` returns a false nil off `terminalErrors`.

## The fix

Port v1's `deleteRunningPipelineIfCurrent` plus its `publishMu`: compare-and-delete, serialized against publication so it is a real CAS rather than a TOCTOU. `csync.Map` has no CAS primitive, which is why the mutex is needed rather than just the comparison.

## Note on the plans

The v0.20 plan's AC 6 for #2806 said the two packages would end up sharing one rule. They now share **publication ordering** but have **diverged on delete discipline** — v1 compares, v2 does not. That divergence should not outlive this issue.

Tier 1. Related: #2806, #2807.

Contributor guide

Open the contributing guide

Research direction

Start in pkg/lifecycle-poc/service.go around lines 1658-1702 and trace recoverPipeline, StartWithBackoff, Start, Set, Stop, and WaitPipeline. Compare the v2 cleanup with v1's fix from #2806, including publication synchronization. Done means recovery cleanup cannot delete a newer running pipeline by key, and live runs remain reachable through Stop and WaitPipeline.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.