ec2tag watcher fails if metadata service is unavailable during a credential refresh
- Dominant language
- Ruby
- Stars
- 2.1k
- Forks
- 248
- PR merge metrics
- No merged PRs in 30d
Description
The `DefaultProvider` takes aws keys from environment variables / config if available, otherwise falling back to ec2 metadata IAM role. If you are using keys from the latter, they refresh automatically when they get near to expiry. If the metadata service is down at this point, Synapse enters a broken state, where it has no credentials and cannot recover. We have seen this in our production environment.
I'm currently looking at a patch whereby I specifically select the `EC2Provider` if no keys are provided by the environment / config, inserting the following into `ec2tag.rb` before the call to `AWS::EC2.new`:
` unless ((@discovery['aws_access_key_id'] || ENV['aws_access_key_id']) \
&& (@discovery['aws_secret_access_key'] || ENV['aws_secret_access_key'] ))
AWS.config(:credential_provider => AWS::Core::CredentialProviders::EC2Provider.new(:retries => 0))
end
`
Does this seem like a reasonable approach? Happy to submit a PR.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in ec2tag.rb at the AWS::EC2.new call and trace how DefaultProvider selects and refreshes credentials. Reproduce the metadata-service outage during refresh, then verify that the watcher can recover usable credentials when the service returns.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, ruby
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100