[JENKINS-71554] EC2ConnectionUpdater is not resestablishing connection

Hi there, we have been noticing this issue where we get the following exception sporadically.

 

hudson.plugins.ec2.EC2Cloud#provision: SlaveTemplate{description='cbci-amznl2-agent', labels='aws-aml2 aws-linux'}. Exception during provisioning

com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 3de0d618-7b0c-4ef1-82f4-8eedf5bd8c88; Proxy: null)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)

    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)

    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)

    at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:34698)

    at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34665)

    at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34654)

    at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeImages(AmazonEC2Client.java:15117)

    at com.amazonaws.services.ec2.AmazonEC2Client.describeImages(AmazonEC2Client.java:15085)

    at hudson.plugins.ec2.SlaveTemplate.getImage(SlaveTemplate.java:1347)

    at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:901)

    at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:717)

    at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:743)

    at hudson.slaves.Cloud.provision(Cloud.java:210)

    at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:726)

    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)

    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)

    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94)

    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)

    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)

    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)

    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

    at java.base/java.lang.Thread.run(Thread.java:829) 

 

In order to unblock ourselves, we have been saving the cloud configuration with no changes, this seems to reestablish the connection to AWS.

 

We have noticed that in some cases, there have been logs from the EC2ConnectionUpdater between instances of the above stack trace.

 

I noticed that the EC2ConnectionUpdater works by catching an AmazonClientException and then triggers the reconnectToEc2 method on EC2Cloud.

 

The only real difference I can see between the calls made to EC2 from this plugin is that the EC2ConnectionUpdater calls describeInstances whereas the SlaveTemplate calls describeImages.

 

If you need any more information from me please feel free to reach out.

 

Thanks,

Alan.

---
Originally reported by a_devine, imported from: EC2ConnectionUpdater is not resestablishing connection

• assignee: raihaan
  
• status: Open
  
• priority: Major
  
• component(s): ec2-plugin
  
• resolution: Unresolved
  
• votes: 4
  
• watchers: 6
  
• imported: 2025-12-06
  

Raw content of original issue

Hi there, we have been noticing this issue where we get the following exception sporadically.


 



hudson.plugins.ec2.EC2Cloud#provision: SlaveTemplate{description='cbci-amznl2-agent', labels='aws-aml2 aws-linux'}. Exception during provisioning

com.amazonaws.services.ec2.model.AmazonEC2Exception: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 3de0d618-7b0c-4ef1-82f4-8eedf5bd8c88; Proxy: null)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1879)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1418)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1387)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1157)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:814)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:781)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:755)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:715)

    at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:697)

    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:561)

    at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:541)

    at com.amazonaws.services.ec2.AmazonEC2Client.doInvoke(AmazonEC2Client.java:34698)

    at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34665)

    at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:34654)

    at com.amazonaws.services.ec2.AmazonEC2Client.executeDescribeImages(AmazonEC2Client.java:15117)

    at com.amazonaws.services.ec2.AmazonEC2Client.describeImages(AmazonEC2Client.java:15085)

    at hudson.plugins.ec2.SlaveTemplate.getImage(SlaveTemplate.java:1347)

    at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:901)

    at hudson.plugins.ec2.EC2Cloud.getNewOrExistingAvailableSlave(EC2Cloud.java:717)

    at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:743)

    at hudson.slaves.Cloud.provision(Cloud.java:210)

    at hudson.slaves.NodeProvisioner$StandardStrategyImpl.apply(NodeProvisioner.java:726)

    at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:325)

    at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:823)

    at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:94)

    at jenkins.security.ImpersonatingScheduledExecutorService$1.run(ImpersonatingScheduledExecutorService.java:69)

    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)

    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)

    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)

    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)

    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

    at java.base/java.lang.Thread.run(Thread.java:829) 



 


In order to unblock ourselves, we have been saving the cloud configuration with no changes, this seems to reestablish the connection to AWS.


 


We have noticed that in some cases, there have been logs from the EC2ConnectionUpdater between instances of the above stack trace.


 


I noticed that the EC2ConnectionUpdater works by catching an AmazonClientException and then triggers the reconnectToEc2 method on EC2Cloud.


 


The only real difference I can see between the calls made to EC2 from this plugin is that the EC2ConnectionUpdater calls describeInstances whereas the SlaveTemplate calls describeImages.


 


If you need any more information from me please feel free to reach out.


 


Thanks,


Alan.

environment

```
Jenkins Version: 2.387.3.5

Jenkins Vender: Cloudbees

EC2 Plugin Version: 2.0.6

Deployment Details:

We have Jenkins controllers running in EKS on one AWS account. EC2 Agents are being deployed in their own account.

We are using an IAM role to handle permissions with an STS token duration of 900 seconds (this was reduced for testing purposes)
```

Start with EC2ConnectionUpdater and EC2Cloud.reconnectToEc2, then compare the updater's describeInstances call with SlaveTemplate.getImage and its describeImages call in the reported provisioning path. Reproduce the RequestExpired failure and trace whether the connection is restored; done means provisioning no longer remains blocked after this failure.

Written by the indexing model from the issue text.