pipe-cd / pipe-cd/pipecd

Lambda drift detection: SubnetIDs comparison fails due to inconsistent sorting

Open
#6,529 4 comments 0 reactions 1 assignee View on GitHub

@rishichirchi is already working on this.

Since Apr 16, 2026.

kind/bug type/bug
Dominant language
Go
Stars
1.4k
Forks
364
Avg merge
1d 4h
Merged PRs (30d)
84

Description

Bug Description

The Lambda drift detector in pkg/app/piped/driftdetector/lambda/detector.go has an inconsistent sorting behavior for SubnetIDs comparison, causing false OUT_OF_SYNC detection and infinite ROLLING_BACK loops.

Root Cause

In the ignoreAndSortParameters() function:

  • The head spec (Git) SubnetIDs are sorted alphabetically
  • The live state (AWS API) SubnetIDs are NOT sorted
  • The code comment states "SubnetIDs are sorted in live state" but this is incorrect — AWS Lambda API does not guarantee sorted SubnetIDs

This means when function.yaml has SubnetIDs in a different order than what AWS returns, PipeCD always detects a diff even though the actual values are identical.

Impact

  • All VPC-attached Lambda functions show perpetual OUT_OF_SYNC
  • SYNC attempts succeed (LAMBDA_SYNC stage SUCCESS) but deployment rolls back because the diff persists
  • This creates an infinite SYNC → ROLLING_BACK → OUT_OF_SYNC loop

Steps to Reproduce

  1. Deploy a Lambda function with VPC config containing 2+ subnets
  2. Ensure function.yaml SubnetIDs order differs from what AWS API returns
  3. Observe that PipeCD drift detector shows OUT_OF_SYNC
  4. Run SYNC — deployment succeeds but status returns to OUT_OF_SYNC

Expected Behavior

Both head spec and live state SubnetIDs should be sorted before comparison (or compared as sets).

Suggested Fix

In ignoreAndSortParameters(), sort live.SubnetIds the same way head.SubnetIds is sorted:

// Current (buggy):
sort.Strings(headSpec.SubnetIds)

// Fix: also sort live state
sort.Strings(headSpec.SubnetIds)
sort.Strings(liveState.SubnetIds)

Environment

  • PipeCD version: v0.50.2 (also confirmed in latest v0.55.0 source)
  • Bug introduced in: PR #5186 (v0.48.9)
  • Platform: Lambda with VPC configuration

Workaround

Sort SubnetIDs alphabetically in function.yaml to match PipeCD's internal sorting.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.