jenkinsci / jenkinsci/ec2-plugin
Request has expired errors when EC2 credential uses IAM role - StaticCredentialsProvider freezes STS session tokens
- Dominant language
- Java
- Stars
- 292
- Forks
- 709
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 4
Description
## Problem
When the EC2 cloud is configured with a `credentialsId` pointing to a Jenkins AWS credential that internally uses `iamRoleArn` (role-based credential), provisioning agents periodically fails with `Request has expired` every ~1 hour:
```
Exception during provisioning
SdkClientException: Request attempt 1 failure: Request has expired.
...
Ec2Exception: Request has expired. (Service: Ec2, Status Code: 400) (SDK Attempt Count: 17)
Reconnecting to EC2 due to RequestExpired or ExpiredToken error
```
After `reconnectToEc2()` the very next call to `describeImages` succeeds immediately.
## Root Cause
When `credentialsId` is set but `roleArn` is **not** set at the EC2 cloud level, `createCredentialsProvider` takes the 2-parameter path:
https://github.com/jenkinsci/ec2-plugin/blob/127f0f109c299eb125a0bd55451f9b55add7a41b/src/main/java/hudson/plugins/ec2/EC2Cloud.java#L1084
```java
AmazonWebServicesCredentials credentials = getCredentials(credentialsId);
if (credentials != null) {
return StaticCredentialsProvider.create(credentials.resolveCredentials());
}
```
`credentials.resolveCredentials()` on a role-based Jenkins credential performs `STS AssumeRole` **once** and returns temporary `AwsSessionCredentials`. These are then **frozen** inside `StaticCredentialsProvider` - no auto-refresh.
After the STS session expires (default 1 hour), all subsequent `describeImages` calls fail. The SDK retries 17 times (due to hardcoded `numRetries(16)`) before throwing, then `reconnectToEc2()` is called which creates a fresh `Ec2Client` with new credentials - hence working immediately after reconnect.
The 5-parameter path with `roleArn` correctly uses `StsAssumeRoleCredentialsProvider` which auto-refreshes:
https://github.com/jenkinsci/ec2-plugin/blob/127f0f109c299eb125a0bd55451f9b55add7a41b/src/main/java/hudson/plugins/ec2/EC2Cloud.java#L1095
## Fix / Workaround
Configure `roleArn` at the EC2 cloud level (not inside the Jenkins credential) and use instance profile or static keys as `credentialsId`. This routes through `StsAssumeRoleCredentialsProvider` with automatic credential refresh:
```yaml
- amazonEC2:
useInstanceProfileForCredentials: true
roleArn: "arn:aws:iam::ACCOUNT:role/ROLE_NAME"
roleSessionName: "Jenkins"
```
## Expected Behavior
If `credentialsId` resolves to a credential containing an IAM role ARN, the resulting STS session credentials should be wrapped in an auto-refreshing provider rather than `StaticCredentialsProvider`.
## Related Issues
- #1901 (JENKINS-71554) - describes similar `RequestExpired` symptoms but focuses on the keepalive path; the root cause reported there overlaps with this issue
- PR #886 - Added `reconnectToEc2()` on `RequestExpired` in `EC2ConnectionUpdater` (partial fix, doesn't address `StaticCredentialsProvider` expiry)
- PR #1008 - Added `ExpiredToken` handling alongside `RequestExpired` in provision catch block
- PR #1120 - Fixed `DefaultCredentialsProvider` being cached as static instance; similar spirit to this issue but different code path
Contributor guide
Research direction
Start in src/main/java/hudson/plugins/ec2/EC2Cloud.java at createCredentialsProvider and compare the credentialsId path with the roleArn path using StsAssumeRoleCredentialsProvider. Done means a Jenkins credential containing an IAM role ARN produces credentials that refresh after the STS session expires instead of remaining in StaticCredentialsProvider, preventing recurring Request has expired failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100