adobe / adobe/shredder

Getting autoScaling information fails fast after rate limit exception

Open
#21 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
17
Forks
9
PR merge metrics
No merged PRs in 30d

Description

### Actual behavior

The shredder start up service fails fast, when unable to fetch information about the instance auto scaling group, due to Rate Limit exception.

### Expected Behavior

The shredder should be able to retry this requests, for a finite amount, in order to avoid leaving instances hanging in auto scaling group

### Steps to reproduce

Ramp up a decent number of instances at the same time, chances to hit a Rate Limit exception goes higher directly with the number of instances ramped up. Usually when trying to ramp up 40 instances, we get around 3-4 instances getting stuck.

StackTrace:
```java
2023-07-27 08:01:23.032 [WrapperSimpleAppMain] ERROR c.a.a.s.c.a.s.AwsAutoScaleGroupHelper - Unable to fetch current AutoScaleGroup for instance: i-070c5880e7ccd1fc4 {}
com.amazonaws.services.autoscaling.model.AmazonAutoScalingException: Rate exceeded (Service: AmazonAutoScaling; Status Code: 400; Error Code: Throttling; Request ID: 0ed76c14-1f08-45a6-a699-fb587016e020)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1658)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1322)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1072)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:745)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:719)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:701)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:669)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:651)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:515)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.doInvoke(AmazonAutoScalingClient.java:3806)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.invoke(AmazonAutoScalingClient.java:3782)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.executeDescribeAutoScalingInstances(AmazonAutoScalingClient.java:1545)
at com.amazonaws.services.autoscaling.AmazonAutoScalingClient.describeAutoScalingInstances(AmazonAutoScalingClient.java:1518)
at com.adobe.aam.shredder.core.aws.servergroup.AwsAutoScaleGroupHelper.getCurrentAutoScalingGroup(AwsAutoScaleGroupHelper.java:40)
at com.adobe.aam.shredder.ec2.aws.InstanceDetailsRetriever.getInstanceDetails(InstanceDetailsRetriever.java:41)
at com.adobe.aam.shredder.ec2.notifier.Notifier.notifyAutoScaleGroup(Notifier.java:36)
at com.adobe.aam.shredder.ec2.service.StartupService.runStartup(StartupService.java:66)
at com.adobe.aam.shredder.ec2.service.StartupService.getStartupResult(StartupService.java:49)
at com.adobe.aam.shredder.ec2.Main.main(Main.java:34)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.tanukisoftware.wrapper.WrapperSimpleApp.run(WrapperSimpleApp.java:325)
at java.lang.Thread.run(Thread.java:748)
2023-07-27 08:01:23.032 [WrapperSimpleAppMain] WARN c.a.a.s.e.a.InstanceDetailsRetriever - Unable to retrieve the Auto Scale Group for the current instance i-070c5880e7ccd1fc4.
```

### Checklist

- [x] Add a descriptive title
- [x] Include explicit steps on how to reproduce the issue
- [ ] Link to a repository that reproduces the issue (**optional**, but **greatly appreciated**)

### Tips

* If possible, create a small repo that reproduces the issue and provide a link
* Don't include full logging output -- link to a gist of the full logging output instead
* Don't paste screenshots of error messages -- copy and paste instead

Contributor guide

Open the contributing guide

Research direction

Start with AwsAutoScaleGroupHelper.getCurrentAutoScalingGroup at AwsAutoScaleGroupHelper.java:40, then follow its callers in InstanceDetailsRetriever.getInstanceDetails and StartupService.runStartup. Reproduce the rate-limit failure by ramping up many instances and verify that a finite retry policy prevents instances from remaining stuck in the Auto Scaling group.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.