Getting autoScaling information fails fast after rate limit exception
- Dominant language
- Shell
- Stars
- 17
- Forks
- 9
- PR merge metrics
- No merged PRs in 30d
Description
### Actual behavior
The shredder start up service fails fast, when unable to fetch information about the instance auto scaling group, due to Rate Limit exception.
### Expected Behavior
The shredder should be able to retry this requests, for a finite amount, in order to avoid leaving instances hanging in auto scaling group
### Steps to reproduce
Ramp up a decent number of instances at the same time, chances to hit a Rate Limit exception goes higher directly with the number of instances ramped up. Usually when trying to ramp up 40 instances, we get around 3-4 instances getting stuck.
StackTrace:
```java
2023-07-27 08:01:23.032 [WrapperSimpleAppMain] ERROR c.a.a.s.c.a.s.AwsAutoScaleGroupHelper - Unable to fetch current AutoScaleGroup for instance: i-070c5880e7ccd1fc4 {}
com.amazonaws.services.autoscaling.model.AmazonAutoScalingException: Rate exceeded (Service: AmazonAutoScaling; Status Code: 400; Error Code: Throttling; Request ID: 0ed76c14-1f08-45a6-a699-fb587016e020)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.doInvoke(AmazonAutoScalingClient.java:3806)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:3782)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.executeDescribeAutoScalingInstances(AmazonAutoScalingClient.java:1545)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.describeAutoScalingInstances(AmazonAutoScalingClient.java:1518)
at com.adobe.aam.shredder.core.aws.servergroup.AwsAutoScaleGroupHelper.getCurrentAutoScalingGroup(AwsAutoScaleGroupHelper.java:40)
at com.adobe.aam.shredder.ec2.aws.InstanceDetailsRetriever.getInstanceDetails(InstanceDetailsRetriever.java:41)
at com.adobe.aam.shredder.ec2.notifier.Notifier.notifyAutoScaleGroup(Notifier.java:36)
at com.adobe.aam.shredder.ec2.service.StartupService.runStartup(StartupService.java:66)
at com.adobe.aam.shredder.ec2.service.StartupService.getStartupResult(StartupService.java:49)
at com.adobe.aam.shredder.ec2.Main.main(Main.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:325)
at java.lang.Thread.run(Thread.java:748)
2023-07-27 08:01:23.032 [WrapperSimpleAppMain] WARN c.a.a.s.e.a.InstanceDetailsRetriever - Unable to retrieve the Auto Scale Group for the current instance i-070c5880e7ccd1fc4.
```
### Checklist
- [x] Add a descriptive title
- [x] Include explicit steps on how to reproduce the issue
- [ ] Link to a repository that reproduces the issue (**optional**, but **greatly appreciated**)
### Tips
* If possible, create a small repo that reproduces the issue and provide a link
* Don't include full logging output -- link to a gist of the full logging output instead
* Don't paste screenshots of error messages -- copy and paste instead
Contributor guide
Research direction
Start with AwsAutoScaleGroupHelper.getCurrentAutoScalingGroup at AwsAutoScaleGroupHelper.java:40, then follow its callers in InstanceDetailsRetriever.getInstanceDetails and StartupService.runStartup. Reproduce the rate-limit failure by ramping up many instances and verify that a finite retry policy prevents instances from remaining stuck in the Auto Scaling group.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100