airbnb / airbnb/synapse

ec2tag watcher needs to back off if Client.RequestLimitExceeded is returned

Open
#198 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
2.1k
Forks
248
PR merge metrics
No merged PRs in 30d

Description

If you have a reasonable number of ec2tag watchers running, you can hit the rate limiter. Rather than a fixed check_interval, there should be a way of doing some kind of backoff (as per the AWS docs). If the Client.RequestLimitExceeded is returned, then there definitely needs to be a backoff - otherwise you just continue to hammer on the door, and the rate limit exceedance will never go away.

I'm yet to discover what the rate limit is, but I hit it today with ~6 servers each watching ~24 ec2tags (plus various other odds and sods running in the background) and the default check_interval

Contributor guide

No contributing guide indexed for this repository

Research direction

Locate the ec2tag watcher and the handling for Client.RequestLimitExceeded, then inspect how the fixed check_interval is applied. Confirm the AWS backoff guidance and existing watcher tests before defining the behavior; done means rate-limit errors cause the watcher to back off instead of repeatedly retrying at the fixed interval.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, ruby
Domain
cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.