aws-cloudformation / aws-cloudformation/cloudformation-coverage-roadmap

AWS::SNS::Topic ReadHandler lacks retry/backoff on ListSubscriptionsByTopic throttling, causing stack failures for topics with many subscriptions

Open
#2,475 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
No language data
Stars
1.1k
Forks
62
PR merge metrics
No merged PRs in 30d

Description

## Description

The `AWS::SNS::Topic` resource handler's `ReadHandler` paginates through all subscriptions via `ListSubscriptionsByTopic` in a tight loop with no delay between API calls. If the topic has a large number of subscriptions (e.g., 1,000+), this generates 10+ rapid paginated calls that can exceed the `ListSubscriptionsByTopic` API rate limit of 30 TPS. When throttled, the handler treats the `ThrottledException` as a hard failure with no retry or exponential backoff, causing the stack operation to fail.

## Current Behavior

During a stack update, the SNS Topic resource handler calls `ListSubscriptionsByTopic` in a `do-while` loop to enumerate all subscriptions. The loop makes consecutive API calls with no delay or rate limiting. If a `ThrottledException` is encountered at any point during pagination, the handler immediately returns `ProgressEvent.failed()` with `HandlerErrorCode.Throttling` — no retry is attempted.

This means a single SNS topic with a large number of subscriptions can cause a stack update to fail, even when no changes are being made to that topic.

## Expected Behavior

The resource handler should implement one or more of the following:
- Exponential backoff and retry on `ThrottledException` during pagination
- Rate limiting / delay between paginated `ListSubscriptionsByTopic` calls
- Leveraging the CloudFormation handler framework's built-in retry/stabilization mechanisms instead of treating throttling as a terminal failure

## Steps to Reproduce

1. Create a stack with an `AWS::SNS::Topic` resource
2. Create 1,000+ subscriptions on that topic (or have deleted subscriptions still appearing in pagination results)
3. Attempt a stack update (even with no changes to the SNS topic)
4. The update fails with: `Unable to retrieve TopicName attribute for AWS::SNS::Topic, with error message Rate exceeded (Service: Sns, Status Code: 400)`

## Impact

- Stack updates fail for any stack containing an SNS topic with a large number of subscriptions
- The failure occurs even when no changes are being made to the topic itself
- Deleted subscriptions that are still in the pagination results (SNS takes ~2 days to purge them) contribute to the subscription count, making this issue worse after bulk subscription cleanup
- AWS Config also makes `ListSubscriptionsByTopic` calls, further contributing to rate limit pressure in the shared per-account quota

## Additional Context

- Affected resource: `AWS::SNS::Topic`
- Handler: `ReadHandler` (also called during update operations)
- `ListSubscriptionsByTopic` returns up to 100 subscriptions per page
- `ListSubscriptionsByTopic` has a hard limit of 30 TPS that cannot be increased

Contributor guide

Open the contributing guide

Research direction

Start at the AWS::SNS::Topic ReadHandler and its do-while pagination over ListSubscriptionsByTopic. Inspect how ThrottledException becomes a failed ProgressEvent and how the CloudFormation handler framework supports retries. Done means throttling during pagination is handled without failing the stack update.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.