airbnb / airbnb/synapse

ec2tag watcher fails if metadata service is unavailable during a credential refresh

Open
#233 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Ruby
Stars
2.1k
Forks
248
PR merge metrics
No merged PRs in 30d

Description

The `DefaultProvider` takes aws keys from environment variables / config if available, otherwise falling back to ec2 metadata IAM role. If you are using keys from the latter, they refresh automatically when they get near to expiry. If the metadata service is down at this point, Synapse enters a broken state, where it has no credentials and cannot recover. We have seen this in our production environment.

I'm currently looking at a patch whereby I specifically select the `EC2Provider` if no keys are provided by the environment / config, inserting the following into `ec2tag.rb` before the call to `AWS::EC2.new`:

` unless ((@discovery['aws_access_key_id'] || ENV['aws_access_key_id']) \
&& (@discovery['aws_secret_access_key'] || ENV['aws_secret_access_key'] ))
AWS.config(:credential_provider => AWS::Core::CredentialProviders::EC2Provider.new(:retries => 0))
end
`

Does this seem like a reasonable approach? Happy to submit a PR.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in ec2tag.rb at the AWS::EC2.new call and trace how DefaultProvider selects and refreshes credentials. Reproduce the metadata-service outage during refresh, then verify that the watcher can recover usable credentials when the service returns.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, ruby
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.