influxdata / influxdata/telegraf

Add support for refreshing AWS Credentials when using temporary session tokens

Open
#13,268 6 comments 0 reactions 0 assignees View on GitHub
feature request upstream
Dominant language
Go
Stars
17.8k
Forks
5.8k
Avg merge
1d 20h
Merged PRs (30d)
161

Description

### Use Case

When using temporary security credentials to connect to AWS, the Telegraf agent should detect changes to the shared_credential_file to prevent errors and the need to restart the agent.

### Expected behavior

The Telegraf agent should refresh the credentials when the shared credential file is updated, periodically or in response to an ExpiredToken error.

It looks like this behaviour was added to the amazon-cloudwatch-agent in the form of a refresh frequency on the credentials provider:

https://github.com/aws/amazon-cloudwatch-agent/blob/main/cfg/aws/refreshable_shared_credentials_provider.go

### Actual behavior

When another process updates the shared_credential_file, the Telegraf agent fails to PutMetricData and instead throws the following error:

`2023-05-11T15:26:09Z E! [outputs.cloudwatch] Unable to write to CloudWatch : operation error CloudWatch: PutMetricData, https response error StatusCode: 403, RequestID: 4d5fbc11-8aba-489a-90b4-0966ab111ae5, api error ExpiredToken: The security token included in the request is expired`

You can reproduce this by configuring the cloudwatch output plugin to use the SSM Managed credential file (e.g. /var/lib/amazon/ssm/credentials) on a non-EC2 host running the SSM Agent:

https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent-technical-details.html#credentials-file
https://docs.aws.amazon.com/systems-manager/latest/userguide/sysman-install-managed-linux.html

### Additional info

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the Telegraf CloudWatch output plugin and trace how it loads the shared_credential_file before PutMetricData. Reproduce the failure with the SSM Managed credential file and an ExpiredToken response, then compare the refreshable provider linked in the issue. Done means updated temporary credentials are used without restarting Telegraf.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.