certificatemanager: deletion of stack with Cognito custom domain fails on CertificateRequestorResource
- Dominant language
- TypeScript
- Stars
- 12.9k
- Forks
- 4.6k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 83
Description
### Describe the bug
Deleting the stack with a custom domain for a Cognito user pool fails on trying to delete `CertificateRequestorResource`.
### Expected Behavior
Deletion succeeds.
### Current Behavior
Deletion fails with message:
> DELETE_FAILED | AWS::CloudFormation::CustomResource | xxxxxxxxxx/CertificateRequestorResource/Default (yyyyyyyyCertificateRequestorResourceF53AA380) Received response status [FAILED] from custom resource. Message returned: Response from describeCertificate did not contain an empty InUseBy list after 10 attempts.
### Reproduction Steps
Deploy the stack which has a Cognito user pool with a custom domain.
Setting such a stack requires defining a certificate for the custom domain. I do it using `DnsValidatedCertificate`, my code (in Clojure with custom helper functions):
```clj
user-pool (-> (UserPool$Builder/create stack "user-pool")
...
(.userPoolName user-pool-name)
.build)
; Cognito requires the parent domain to have a valid DNS A record.
; The parent may be the root of the domain, or a child domain that is one step up in the domain hierarchy.
; For example, if your custom domain is auth.xyz.example.com,
; Cognito must be able to resolve xyz.example.com to an IP address.
;
; The record points "nowhere", https://stackoverflow.com/questions/51249583.
apex (dns/domain user-pool-name "foo.com")
_ (cdk.route53/add-a-record stack apex (RecordTarget/fromIpAddresses (into-array ["127.0.0.1"])))
user-pool-domain (dns/domain "auth" apex)
hosted-zone (cdk.route53/memoized-fetch-hosted-zone stack user-pool-domain)
cert (-> (DnsValidatedCertificate$Builder/create stack (str user-pool-domain "-cert"))
(.domainName user-pool-domain)
(.hostedZone hosted-zone)
(.region (str Region/US_EAST_1)) ; This region is required by Cognito
.build)
domain (.addDomain user-pool "domain" (-> (UserPoolDomainOptions/builder)
(.customDomain (-> (CustomDomainOptions/builder)
(.certificate cert)
(.domainName user-pool-domain)
.build))
.build))
_ (cdk.route53/add-a-record stack user-pool-domain (RecordTarget/fromAlias (UserPoolDomainTarget. domain)))
```
### Possible Solution
_No response_
### Additional Information/Context
_No response_
### CDK CLI Version
2.100.0 (build e1b5c77)
### Framework Version
2.100.0
### Node.js Version
18.17.1
### OS
macOS
### Language
Java
### Language Version
Java (17)
### Other information
### Cause
The cause seems to be that the certificate is still used by the "phantom" CloudFront distribution which belongs to the unknown account 455458493081 and I can't find it anywhere in the GUI.
It can be seen in the ACM GUI or via `aws acm describe-certificate --certificate-arn ... --region us-east-1` and then looking at `InUseBy` key.
After a few minutes this dependency is automatically cleaned and the repeated attempt to delete the stack will succeed after that.
I suspect this is the distribution containing the Cognito's hosted UI website.
I found a single mention of the similar Cognito problem in https://stackoverflow.com/questions/75134728/phantom-cloudfront-distribution-blocks-me-from-creating-cognito-custom-domain. And the answer there states:
> The CloudFront distribution that AWS creates for the custom Cognito domain will be removed in a few hours after you delete the user pool (or delete the custom domain via the Cognito console / API). This seems to be completely hidden from the user (you).
But there are several reports of a similar issue with certificates for API Gateway, e.g.:
* https://forum.serverless.com/t/solved-cannot-delete-certificate-cloudfront-distribution-not-showing/3792
* https://www.reddit.com/r/aws/comments/xn6r2e/how_to_remove_all_usages_of_an_acm_certificate/
* https://repost.aws/questions/QU5RI-7lr2R5m_oxe8N6BTRQ/i-can-t-delete-my-certificate-because-it-s-associated-with-an-invisible-cloudfront-distribution
### Workaround attempt
I tried to retain the certificate on deletion via `(.applyRemovalPolicy cert RemovalPolicy/RETAIN_ON_UPDATE_OR_DELETE)`. This allows the stack deletion to succeed. But when I deployed the same stack again immediately it failed with:
> user-pool/domain (userpooldomainB4026A3C) One or more of the CNAMEs you provided are already associated with a different resource. (Service: AmazonCloudFront; Status Code: 409; Error Code: CNAMEAlreadyExists; Request ID: 6e3993dc-5cb7-4d0d-a267-ed58ca49dee3; Proxy: null) (Service: AWSCognitoIdentityProviderService; Status Code: 400; Error Code: InvalidParameterException; Request ID: 562d0982-bc92-4923-93d1-55732114571f; Proxy: null)
Strangely, deploying one more time succeeded. But in any case, it doesn't seem to be a reliable workaround and with time will pollute ACM with unused certificates.
### Solution ideas
1) The ideal solution is to fix it somewhere in CloudFront or Cognito. So that deletion of the pool immediately cleans the corresponding certificate `InUseBy` array.
1) The solution in CDK could be to increase the number of attempts in `aws-certificatemanager/dns-validated-certificate-handler` `deleteCertificate` function: https://github.com/aws/aws-cdk/blob/c66e197f6f8840da6475383dbf2421c3b06ea417/packages/%40aws-cdk/custom-resource-handlers/lib/aws-certificatemanager/dns-validated-certificate-handler/index.js#L160
Contributor guide
Research direction
Start by reading packages/@aws-cdk/custom-resource-handlers/lib/aws-certificatemanager/dns-validated-certificate-handler/index.js around deleteCertificate and inspect how the InUseBy response is retried. Reproduce the Cognito custom-domain stack deletion and use aws acm describe-certificate in us-east-1 to observe when the hidden CloudFront dependency clears; done means deletion no longer fails while that cleanup is pending.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, typescript
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100