jenkinsci / jenkinsci/ec2-plugin

Request has expired errors when EC2 credential uses IAM role - StaticCredentialsProvider freezes STS session tokens

Open
#2,011 0 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
292
Forks
709
Avg merge
3d 16h
Merged PRs (30d)
4

Description

## Problem

When the EC2 cloud is configured with a `credentialsId` pointing to a Jenkins AWS credential that internally uses `iamRoleArn` (role-based credential), provisioning agents periodically fails with `Request has expired` every ~1 hour:

```
Exception during provisioning
SdkClientException: Request attempt 1 failure: Request has expired.
...
Ec2Exception: Request has expired. (Service: Ec2, Status Code: 400) (SDK Attempt Count: 17)

Reconnecting to EC2 due to RequestExpired or ExpiredToken error
```

After `reconnectToEc2()` the very next call to `describeImages` succeeds immediately.

## Root Cause

When `credentialsId` is set but `roleArn` is **not** set at the EC2 cloud level, `createCredentialsProvider` takes the 2-parameter path:

https://github.com/jenkinsci/ec2-plugin/blob/127f0f109c299eb125a0bd55451f9b55add7a41b/src/main/java/hudson/plugins/ec2/EC2Cloud.java#L1084

```java
AmazonWebServicesCredentials credentials = getCredentials(credentialsId);
if (credentials != null) {
return StaticCredentialsProvider.create(credentials.resolveCredentials());
}
```

`credentials.resolveCredentials()` on a role-based Jenkins credential performs `STS AssumeRole` **once** and returns temporary `AwsSessionCredentials`. These are then **frozen** inside `StaticCredentialsProvider` - no auto-refresh.

After the STS session expires (default 1 hour), all subsequent `describeImages` calls fail. The SDK retries 17 times (due to hardcoded `numRetries(16)`) before throwing, then `reconnectToEc2()` is called which creates a fresh `Ec2Client` with new credentials - hence working immediately after reconnect.

The 5-parameter path with `roleArn` correctly uses `StsAssumeRoleCredentialsProvider` which auto-refreshes:

https://github.com/jenkinsci/ec2-plugin/blob/127f0f109c299eb125a0bd55451f9b55add7a41b/src/main/java/hudson/plugins/ec2/EC2Cloud.java#L1095

## Fix / Workaround

Configure `roleArn` at the EC2 cloud level (not inside the Jenkins credential) and use instance profile or static keys as `credentialsId`. This routes through `StsAssumeRoleCredentialsProvider` with automatic credential refresh:

```yaml
- amazonEC2:
useInstanceProfileForCredentials: true
roleArn: "arn:aws:iam::ACCOUNT:role/ROLE_NAME"
roleSessionName: "Jenkins"
```

## Expected Behavior

If `credentialsId` resolves to a credential containing an IAM role ARN, the resulting STS session credentials should be wrapped in an auto-refreshing provider rather than `StaticCredentialsProvider`.

## Related Issues

- #1901 (JENKINS-71554) - describes similar `RequestExpired` symptoms but focuses on the keepalive path; the root cause reported there overlaps with this issue
- PR #886 - Added `reconnectToEc2()` on `RequestExpired` in `EC2ConnectionUpdater` (partial fix, doesn't address `StaticCredentialsProvider` expiry)
- PR #1008 - Added `ExpiredToken` handling alongside `RequestExpired` in provision catch block
- PR #1120 - Fixed `DefaultCredentialsProvider` being cached as static instance; similar spirit to this issue but different code path

Contributor guide

Open the contributing guide

Research direction

Start in src/main/java/hudson/plugins/ec2/EC2Cloud.java at createCredentialsProvider and compare the credentialsId path with the roleArn path using StsAssumeRoleCredentialsProvider. Done means a Jenkins credential containing an IAM role ARN produces credentials that refresh after the STS session expires instead of remaining in StaticCredentialsProvider, preventing recurring Request has expired failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.