influxdata / influxdata/telegraf
Add support for refreshing AWS Credentials when using temporary session tokens
- Dominant language
- Go
- Stars
- 17.8k
- Forks
- 5.8k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 161
Description
### Use Case
When using temporary security credentials to connect to AWS, the Telegraf agent should detect changes to the shared_credential_file to prevent errors and the need to restart the agent.
### Expected behavior
The Telegraf agent should refresh the credentials when the shared credential file is updated, periodically or in response to an ExpiredToken error.
It looks like this behaviour was added to the amazon-cloudwatch-agent in the form of a refresh frequency on the credentials provider:
https://github.com/aws/amazon-cloudwatch-agent/blob/main/cfg/aws/refreshable_shared_credentials_provider.go
### Actual behavior
When another process updates the shared_credential_file, the Telegraf agent fails to PutMetricData and instead throws the following error:
`2023-05-11T15:26:09Z E! [outputs.cloudwatch] Unable to write to CloudWatch : operation error CloudWatch: PutMetricData, https response error StatusCode: 403, RequestID: 4d5fbc11-8aba-489a-90b4-0966ab111ae5, api error ExpiredToken: The security token included in the request is expired`
You can reproduce this by configuring the cloudwatch output plugin to use the SSM Managed credential file (e.g. /var/lib/amazon/ssm/credentials) on a non-EC2 host running the SSM Agent:
https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent-technical-details.html#credentials-file
https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-managed-linux.html
### Additional info
_No response_
Contributor guide
Research direction
Start with the Telegraf CloudWatch output plugin and trace how it loads the shared_credential_file before PutMetricData. Reproduce the failure with the SSM Managed credential file and an ExpiredToken response, then compare the refreshable provider linked in the issue. Done means updated temporary credentials are used without restarting Telegraf.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100