aws / aws/agentcore-cli

OAuth2 credential providers leak on teardown (CUSTOM_JWT harness / OAuth gateway) — exhausts 50-provider account quota

Open
#1,673 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
TypeScript
Stars
283
Forks
95
Avg merge
1d 2h
Merged PRs (30d)
183

Description

## Description

`agentcore deploy` registers a **managed OAuth2 credential provider** as an imperative pre-deploy step (outside the CloudFormation stack) whenever a CUSTOM_JWT harness — or an OAuth-outbound gateway — is configured with `--client-id/--client-secret`. Tearing the project down with `agentcore remove all` + `agentcore deploy` (or `remove` of the individual resource) destroys the CloudFormation stack but **never deletes the OAuth2 credential provider**. The provider is orphaned in the customer's account.

Each AWS account is capped at **50 OAuth2 credential providers** (Service Quotas code `L-431051DC`). Because nothing reaps them, repeated create/teardown cycles accumulate orphaned providers until the quota is exhausted, after which **every** subsequent CUSTOM_JWT/OAuth deploy fails with:

```
The number of agent identity Oauth2 credential providers in this account has reached its limit
```

This is the same root cause that recently broke the E2E suite (`harness-custom-jwt.test.ts`); that side was patched with test-scoped cleanup, but the **underlying product behavior — orphaning OAuth2 providers on teardown — is unfixed for real users.**

## Steps to Reproduce

1. `agentcore create --name MyProj --no-agent`
2. `agentcore add harness --name MyProj --authorizer-type CUSTOM_JWT --discovery-url --allowed-clients --client-id --client-secret ` (registers a managed OAuth credential named `MyProj-oauth`)
3. `agentcore deploy --yes` → a `MyProj-oauth` OAuth2 credential provider is created in AgentCore Identity
4. `agentcore remove all && agentcore deploy --yes` (teardown) → CloudFormation stack is destroyed
5. **Observe:** `aws bedrock-agentcore-control list-oauth2-credential-providers` still shows `MyProj-oauth`. It was never deleted.
6. Repeat the create/deploy/teardown cycle ~50 times (or share an account across many projects/CI) → every CUSTOM_JWT deploy now fails with the quota-limit error above.

## Expected Behavior

Teardown (`remove all` + teardown deploy, and individual resource `remove`) should delete the managed OAuth2 credential provider it created during deploy — the same way payment credential providers are already reaped — so no resource is orphaned in the customer's account.

## Actual Behavior

The OAuth2 credential provider is leaked permanently. Stack destroy does not touch it because it is created imperatively, outside the stack.

## Root Cause

OAuth2 credential providers are created imperatively pre-deploy:

- `src/cli/operations/deploy/pre-deploy-identity.ts` → `setupOAuth2Providers` → `createOAuth2Provider`
- `src/cli/operations/identity/oauth2-credential-provider.ts` → `CreateOauth2CredentialProviderCommand`
- Managed credential name = `computeManagedOAuthCredentialName(resourceName)` → `` `${name}-oauth` `` (`src/cli/primitives/credential-utils.ts`)

But the teardown path only cleans up **payment** credential providers — there is no OAuth2 equivalent:

```ts
// src/cli/commands/deploy/actions.ts (~L536)
if (context.isTeardownDeploy) {
// Clean up imperative payment credential providers (CFN stack delete handles manager/connector/roles).
// Harnesses are part of the CloudFormation stack, so stack destroy handles them. <-- misleading
...
await cleanupPaymentCredentialProviders({ region: target.region, payments: existingPayments });
...
const teardown = await performStackTeardown(target.name);
}
```

The comment "Harnesses are part of the CloudFormation stack, so stack destroy handles them" is true for the harness *runtime* resource but **not** for its managed OAuth2 credential provider, which lives outside the stack. A grep confirms **nothing** in `src/` calls `DeleteOauth2CredentialProvider` — only ApiKey and Payment providers have delete paths.

## Proposed Fix

Add OAuth2 credential provider cleanup to the teardown path, mirroring `cleanupPaymentCredentialProviders`:

1. Add a `deleteOAuth2Provider` / `cleanupOAuth2CredentialProviders` operation in `src/cli/operations/deploy/pre-deploy-identity.ts` (or alongside the existing OAuth2 ops in `src/cli/operations/identity/oauth2-credential-provider.ts`) using `DeleteOauth2CredentialProviderCommand`.
2. Call it from the `isTeardownDeploy` branch in `src/cli/commands/deploy/actions.ts`, iterating the project's `OAuthCredentialProvider` credentials (resolving names via `computeManagedOAuthCredentialName` for managed harness/gateway creds). Make it best-effort like payment cleanup (warn, don't fail teardown).
3. Consider the same on individual-resource `remove` for a harness/gateway that owns a managed OAuth credential, so a single `remove` doesn't orphan the provider either.
4. Be careful not to delete a provider that another still-deployed resource in the project references (only reap providers owned by the resource(s) being torn down).

## Acceptance Criteria

- [ ] After `agentcore remove all` + teardown deploy of a CUSTOM_JWT harness project, `list-oauth2-credential-providers` shows the `-oauth` provider is gone.
- [ ] Removing an individual harness/gateway that owns a managed OAuth credential deletes its OAuth2 provider.
- [ ] Cleanup is best-effort (a delete failure logs a warning and does not abort teardown).
- [ ] Cleanup never deletes a provider still referenced by another deployed resource in the project.
- [ ] Test coverage for the teardown cleanup path.

## Notes

- Quota `L-431051DC` ("Resource OAuth2 credential providers", default 50) is adjustable, but a limit increase only delays the leak — the durable fix is reaping on teardown.
- Related (test-side mitigation, separate PR): added `cleanupStaleOAuth2CredentialProviders` to the E2E `globalSetup` hook and per-test teardown in `harness-custom-jwt.test.ts`. That keeps CI green but does not fix the product behavior described here.

Contributor guide

Open the contributing guide

Research direction

Start in src/cli/commands/deploy/actions.ts at the isTeardownDeploy branch, then compare cleanupPaymentCredentialProviders in src/cli/operations/deploy/pre-deploy-identity.ts with the OAuth2 operations in src/cli/operations/identity/oauth2-credential-provider.ts. Trace managed names through src/cli/primitives/credential-utils.ts and the individual-resource remove path; done means safe best-effort cleanup, preserved shared providers, and teardown tests covering the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, typescript
Domain
cli, cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.