OAuth2 credential providers leak on teardown (CUSTOM_JWT harness / OAuth gateway) — exhausts 50-provider account quota
- Dominant language
- TypeScript
- Stars
- 283
- Forks
- 95
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 183
Description
## Description
`agentcore deploy` registers a **managed OAuth2 credential provider** as an imperative pre-deploy step (outside the CloudFormation stack) whenever a CUSTOM_JWT harness — or an OAuth-outbound gateway — is configured with `--client-id/--client-secret`. Tearing the project down with `agentcore remove all` + `agentcore deploy` (or `remove` of the individual resource) destroys the CloudFormation stack but **never deletes the OAuth2 credential provider**. The provider is orphaned in the customer's account.
Each AWS account is capped at **50 OAuth2 credential providers** (Service Quotas code `L-431051DC`). Because nothing reaps them, repeated create/teardown cycles accumulate orphaned providers until the quota is exhausted, after which **every** subsequent CUSTOM_JWT/OAuth deploy fails with:
```
The number of agent identity Oauth2 credential providers in this account has reached its limit
```
This is the same root cause that recently broke the E2E suite (`harness-custom-jwt.test.ts`); that side was patched with test-scoped cleanup, but the **underlying product behavior — orphaning OAuth2 providers on teardown — is unfixed for real users.**
## Steps to Reproduce
1. `agentcore create --name MyProj --no-agent`
2. `agentcore add harness --name MyProj --authorizer-type CUSTOM_JWT --discovery-url --allowed-clients --client-id --client-secret ` (registers a managed OAuth credential named `MyProj-oauth`)
3. `agentcore deploy --yes` → a `MyProj-oauth` OAuth2 credential provider is created in AgentCore Identity
4. `agentcore remove all && agentcore deploy --yes` (teardown) → CloudFormation stack is destroyed
5. **Observe:** `aws bedrock-agentcore-control list-oauth2-credential-providers` still shows `MyProj-oauth`. It was never deleted.
6. Repeat the create/deploy/teardown cycle ~50 times (or share an account across many projects/CI) → every CUSTOM_JWT deploy now fails with the quota-limit error above.
## Expected Behavior
Teardown (`remove all` + teardown deploy, and individual resource `remove`) should delete the managed OAuth2 credential provider it created during deploy — the same way payment credential providers are already reaped — so no resource is orphaned in the customer's account.
## Actual Behavior
The OAuth2 credential provider is leaked permanently. Stack destroy does not touch it because it is created imperatively, outside the stack.
## Root Cause
OAuth2 credential providers are created imperatively pre-deploy:
- `src/cli/operations/deploy/pre-deploy-identity.ts` → `setupOAuth2Providers` → `createOAuth2Provider`
- `src/cli/operations/identity/oauth2-credential-provider.ts` → `CreateOauth2CredentialProviderCommand`
- Managed credential name = `computeManagedOAuthCredentialName(resourceName)` → `` `${name}-oauth` `` (`src/cli/primitives/credential-utils.ts`)
But the teardown path only cleans up **payment** credential providers — there is no OAuth2 equivalent:
```ts
// src/cli/commands/deploy/actions.ts (~L536)
if (context.isTeardownDeploy) {
// Clean up imperative payment credential providers (CFN stack delete handles manager/connector/roles).
// Harnesses are part of the CloudFormation stack, so stack destroy handles them. <-- misleading
...
await cleanupPaymentCredentialProviders({ region: target.region, payments: existingPayments });
...
const teardown = await performStackTeardown(target.name);
}
```
The comment "Harnesses are part of the CloudFormation stack, so stack destroy handles them" is true for the harness *runtime* resource but **not** for its managed OAuth2 credential provider, which lives outside the stack. A grep confirms **nothing** in `src/` calls `DeleteOauth2CredentialProvider` — only ApiKey and Payment providers have delete paths.
## Proposed Fix
Add OAuth2 credential provider cleanup to the teardown path, mirroring `cleanupPaymentCredentialProviders`:
1. Add a `deleteOAuth2Provider` / `cleanupOAuth2CredentialProviders` operation in `src/cli/operations/deploy/pre-deploy-identity.ts` (or alongside the existing OAuth2 ops in `src/cli/operations/identity/oauth2-credential-provider.ts`) using `DeleteOauth2CredentialProviderCommand`.
2. Call it from the `isTeardownDeploy` branch in `src/cli/commands/deploy/actions.ts`, iterating the project's `OAuthCredentialProvider` credentials (resolving names via `computeManagedOAuthCredentialName` for managed harness/gateway creds). Make it best-effort like payment cleanup (warn, don't fail teardown).
3. Consider the same on individual-resource `remove` for a harness/gateway that owns a managed OAuth credential, so a single `remove` doesn't orphan the provider either.
4. Be careful not to delete a provider that another still-deployed resource in the project references (only reap providers owned by the resource(s) being torn down).
## Acceptance Criteria
- [ ] After `agentcore remove all` + teardown deploy of a CUSTOM_JWT harness project, `list-oauth2-credential-providers` shows the `-oauth` provider is gone.
- [ ] Removing an individual harness/gateway that owns a managed OAuth credential deletes its OAuth2 provider.
- [ ] Cleanup is best-effort (a delete failure logs a warning and does not abort teardown).
- [ ] Cleanup never deletes a provider still referenced by another deployed resource in the project.
- [ ] Test coverage for the teardown cleanup path.
## Notes
- Quota `L-431051DC` ("Resource OAuth2 credential providers", default 50) is adjustable, but a limit increase only delays the leak — the durable fix is reaping on teardown.
- Related (test-side mitigation, separate PR): added `cleanupStaleOAuth2CredentialProviders` to the E2E `globalSetup` hook and per-test teardown in `harness-custom-jwt.test.ts`. That keeps CI green but does not fix the product behavior described here.
Contributor guide
Research direction
Start in src/cli/commands/deploy/actions.ts at the isTeardownDeploy branch, then compare cleanupPaymentCredentialProviders in src/cli/operations/deploy/pre-deploy-identity.ts with the OAuth2 operations in src/cli/operations/identity/oauth2-credential-provider.ts. Trace managed names through src/cli/primitives/credential-utils.ts and the individual-resource remove path; done means safe best-effort cleanup, preserved shared providers, and teardown tests covering the acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cli, cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100