aws / aws/aws-codedeploy-agent

ExpiredTokenException error on credential file refresh on on-premise (/root/.aws/credentials)

Open
#413 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Rust
Stars
337
Forks
195
Avg merge
10h 54m
Merged PRs (30d)
4

Description

### Env
- On-premise
- Use AWS Systems Manager(SSM) Hybrid Activation
- SSM agent updates /root/.aws/credentials every 30 minutes for new credentials

### Configuration
/etc/codedeploy-agent/conf/codedeploy.onpremises.yml
```
iam_session_arn: arn:aws:sts:::assumed-role/AWSOnpremServerRole/AWSOnpremServerRole_CDP
aws_credentials_file: /root/.aws/credentials
region: ap-northeast-2
```

### Issue
At the first time, when codedeploy-agent start no credential issue, but if the first credential updated by SSM agent expired then codedeploy-agent get error even though credentials file is updated.
```
2026-02-11T07:01:28 ERROR [codedeploy-agent(5470)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Cannot reach InstanceService: Aws::CodeDeployCommand::Errors::ExpiredTokenException - The security token included in the request is expired
2026-02-11T07:01:28 DEBUG [codedeploy-agent(5470)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Sleeping 7 seconds.
.......
2026-02-11T07:01:38 INFO [codedeploy-agent(5470)]: [Aws::CodeDeployCommand::Client 400 2.126136 3 retries] poll_host_command(host_identifier:"arn:aws:sts::558846430793:assumed-role/AWSOnpremServerRole/mi-0c9c2ec582198650e") Aws::CodeDeployCommand::Errors::ExpiredTokenException The security token included in the request is expired
2026-02-11T07:01:38 ERROR [codedeploy-agent(5470)]: InstanceAgent::Plugins::CodeDeployPlugin::CommandPoller: Error polling for host commands: Aws::CodeDeployCommand::Errors::ExpiredTokenException - The security token included in the request is expired - /usr/local/share/gems/gems/aws-sdk-core-3.242.0/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call'
```
However, if the aws_credentials_file path is not `/root/.aws/credentials`, credential is updated and no error.

### Reason
[`file_credentials.rb`](https://github.com/aws/aws-codedeploy-agent/blob/main/lib/instance_agent/file_credentials.rb)
```
def refresh
@credentials = Aws::SharedCredentials.new(path: @path).credentials
raise "Failed to load credentials from path #{@path}" if @credentials.nil?
@expiration = Time.new + 1800
rescue Aws::Errors::NoSuchProfileError
raise "Failed to load credentials from path #{@path}"
end
```

[`aws-sdk-core/shared_credentials.rb`](https://github.com/aws/aws-sdk-ruby/blob/version-3/gems/aws-sdk-core/lib/aws-sdk-core/shared_credentials.rb)
```
def initialize(options = {})
shared_config = Aws.shared_config
@path = options[:path]
@path ||= shared_config.credentials_path
@profile_name = options[:profile_name]
@profile_name ||= ENV['AWS_PROFILE']
@profile_name ||= shared_config.profile_name
if @path && @path == shared_config.credentials_path
@credentials = shared_config.credentials(profile: @profile_name)
else
config = SharedConfig.new(
credentials_path: @path,
profile_name: @profile_name
)
@credentials = config.credentials(profile: @profile_name)
end
@metrics = ['CREDENTIALS_CODE']
end
```

I think when the credential path is `/root/.aws/credentials`, the code uses the cached singleton `shared_config` object, which prevents credentials from being refreshed even when the file is updated.

### Solution
1. Update document
[In this document](https://docs.aws.amazon.com/codedeploy/latest/userguide/register-on-premises-instance-iam-session-arn.html#register-on-premises-instance-iam-session-arn-3), It should be explicitly stated that it is recommended to use a path other than /root/.aws/credentials for the credentials-file path.

2. Update code
Use credentials as usual when not on the default path. However, it is true that the credentials for that path do not generally change, and since the current situation is an exception, it would be better to modify the document as in 1.
```
else
config = SharedConfig.new(
credentials_path: @path,
profile_name: @profile_name
)
@credentials = config.credentials(profile: @profile_name)
end
```

---

If there is anything I misunderstood, please let me know.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.