pipe-cd / pipe-cd/pipecd

Older live-state event can silently overwrite a newer one in the same batch

Open
#7,136 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
1.4k
Forks
364
Avg merge
1d 4h
Merged PRs (30d)
84

Description

What's happening

PatchKubernetesApplicationLiveState (pkg/app/server/applicationlivestatestore/store.go) applies a batch of Kubernetes resource-state events to a snapshot. Each event carries a version, and the code is supposed to skip any event older than what's already stored:

if ev.SnapshotVersion.IsBefore(*snapshot.Version) {
    continue
}

The problem: snapshot.Version is fetched once at the top of the function and never updated while the loop applies events. So every event in the batch gets compared against that same original version, not against the version the previous event in the loop just moved the state to.

Why it matters

If a batch has two events for the same resource where an older one arrives after a newer one (a redelivered retry is a normal way this can happen), the check above doesn't catch it. The older event silently overwrites the newer state — no error, no log.

How I found it

Traced how piped assigns event versions (pkg/app/piped/livestatestore/kubernetes/appnodes.go) through to how the server applies them. Wrote a test with two out-of-order events for the same resource — the final state ends up on the stale value.

I have a fix and a test ready, opening a PR for it.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in pkg/app/server/applicationlivestatestore/store.go and trace the event versions from pkg/app/piped/livestatestore/kubernetes/appnodes.go. Run or reproduce the out-of-order two-event test described in the issue. Done means an older event in the same batch cannot replace a newer state, with regression coverage for that ordering.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
backend, devops, infrastructure
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.