aws / aws/aws-cdk

(aws-codebuild): orphan webhook record blocks GITLAB_SELF_MANAGED project recreation after CFN rollback

Open
#38,019 2 comments 0 reactions 0 assignees View on GitHub
@aws-cdk/aws-codebuild effort/small p3
Dominant language
TypeScript
Stars
12.9k
Forks
4.6k
Avg merge
2d 3h
Merged PRs (30d)
83

Description

## Issue

`AWS::CodeBuild::Webhook` with `triggers.scopeConfiguration.scope: GITLAB_GROUP` against a `GITLAB_SELF_MANAGED` source: a CFN deploy that successfully creates the webhook before failing on a sibling resource leaves an orphan record in AWS's internal CodeBuild webhook store during rollback. Subsequent `CreateWebhook` against the same `(account, region, projectName, scopeConfiguration)` tuple fails with:

```
Failed to call CreateWebhook, reason: GitLab Runner webhook already exists for repo and project name
(Service: AWSCodeBuild; Status Code: 400; Error Code: InvalidInputException)
```

The orphan record persists indefinitely (many hours empirically; AWS docs document no TTL). GitLab-side analog: [hashicorp/terraform-provider-aws#30551](https://github.com/hashicorp/terraform-provider-aws/issues/30551) — same shape but on the source provider's side rather than AWS's internal table.

## Reproduction

1. CDK stack with multiple `CfnProject` resources, each with a `CfnWebhook` configured for `scope: GITLAB_GROUP`. `CODECONNECTIONS` auth.
2. Deploy. If any non-first project in the stack fails (transient AWS error, dependency timing, anything), CFN rolls back. The successfully-created webhooks orphan their project names on AWS's side during the rollback cleanup.
3. Redeploy unchanged. Previously-successful project names now fail with `GitLab Runner webhook already exists for repo and project name`.

## Behavior verified

The orphan is in an AWS-internal store with no customer API surface:

| Operation | Behavior |
|---|---|
| `aws codebuild describe-project --name ` | Project does not exist |
| `aws codebuild update-webhook --project-name ` | `Webhook does not exist for this project` |
| `aws codebuild delete-webhook --project-name ` | `Project does not exist` |
| `aws codebuild create-webhook --project-name ` | `already exists` (this is the bug) |
| `aws codebuild create-webhook --project-name --manual-creation` | Succeeds (the orphan check fires only in the auto-creation code path) |
| `aws codeconnections get-connection-token` | Not a public API; CLI returns `Found invalid choice 'get-connection-token'` |
| `aws codebuild update-webhook` flag to flip `--manual-creation`/`--no-manual-creation` | Not exposed — `update-webhook` accepts only branch-filter, rotate-secret, filter-groups, build-type, pull-request-build-policy. Manual webhooks stay manual until deleted. |
| CloudFormation IMPORT of `AWS::CodeBuild::Webhook` | Orphan not visible to standard describe APIs; cannot import |

Recreating the CodeConnections `Connection` (new ARN, fresh OAuth grant) does NOT clear the orphan. Recreating the parent `Host` (new ARN, brand-new GitLab OAuth application — the prior app no longer exists on GitLab) does NOT clear the orphan either. The orphan key includes only `(account, region, projectName, scopeConfiguration)`; no Host or Connection ARN component.

**Empirically verified that normal (non-rollback) deploy/destroy is clean.** `cdk deploy` of a fresh-named project + webhook, then `cdk destroy`, then a third `cdk deploy` of the same name all succeed cleanly. The orphan accumulation is *specifically* the CFN rollback cleanup path; the customer-initiated `cdk destroy` path properly cleans up AWS's webhook record.

## Mitigations

1. **Generation-suffix the project name**. Pick a stable suffix (e.g., `-0`) on every CodeBuild runner project name, set via a single shared constant in CDK config and a matching workflow variable in your CI YAML so the runner-tag prefix tracks the project name. When a name gets squatted, bump the constant in both places — CDK updates rebuild all projects under fresh names, old names stay invisible in AWS's DB forever, no further surgery. This is the cleanest pure-OAuth workaround.

2. **`ManualCreation: true`** on `AWS::CodeBuild::Webhook`. Bypasses the orphan check entirely AND prevents new orphans from accumulating (AWS doesn't call the source-provider API, so AWS has no internal state to leave dangling). Cost: per-project source-side webhook setup (operator or CDK custom resource).

3. **AWS Support case**. Appears to be the only path to *actually delete* the orphan records — the implication being there's internal admin tooling for it.

## Ask

- Documentation: is there a documented or empirical TTL for these records?
- API surface: any planned operation that lets customers list and delete orphan webhook records by project name without a corresponding CodeBuild project?
- Could the `CreateWebhook` auto-path's duplicate check be made consistent with `DeleteWebhook`'s view? Right now they diverge on the same name (`Create` sees the orphan, `Delete` doesn't see anything).

## Why filed here

The bug surfaces through `aws-cdk-lib`'s `aws_codebuild` constructs but the root cause is in the AWS CodeBuild service. Filing here because (a) this is where CDK users searching for the error will find documentation, and (b) maintainers here can route to the service team.

## Environment

- `aws-cdk-lib`: ^2.234.x
- AWS region: us-east-2 (reproducible in any region; orphan key is region-scoped)
- Source type: `GITLAB_SELF_MANAGED` via CodeConnections (Self-Managed runner GA'd Feb 2025)
- Scope: `GITLAB_GROUP` (group hooks; requires GitLab Premium)

Contributor guide

Open the contributing guide

Research direction

Start with the CDK CfnProject/CfnWebhook reproduction and compare the listed AWS CodeBuild create-webhook, delete-webhook, and describe-project operations during CloudFormation rollback. The issue does not identify a repository file or test; done would require an AWS service-level fix or documented TTL and customer cleanup API for orphan records.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.