jenkinsci / jenkinsci/ec2-plugin
[JENKINS-64520] EC2 node not start after stop/disconnect with parameter Idle termination time
- Dominant language
- Java
- Stars
- 292
- Forks
- 709
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 4
Description
We have a problem - after stopping ec2, the nodes do not start again. Playable on three jenkins servers. One of them is clean with plugin Amazon EC2.
Plugin settings and access to ec2 are the same
Common settings on plugin EC2
Idle termination time: 30
Stop/Disconnect on Idle Timeout: yes
Minimum number of instances: 0
Minimum number of spare instances: 0
Host key Verification Strategy: Off
Connection stratagy: PublicDNS
When I checked credentials and AMI - plugin says - success
Logs on one node when I launch
2020-12-28 12:33:18.949+0000 [id=25] INFO h.p.ec2.EC2RetentionStrategy#internalCheck: Idle timeout of EC2 (Test Amazon connection) - Jenkins PHP70 (same-instance-id) after 30 idle minutes, instance statusRUNNING
2020-12-28 12:33:18.950+0000 [id=25] INFO h.plugins.ec2.EC2AbstractSlave#idleTimeout: EC2 instance idle time expired: same-instance-id
2020-12-28 12:33:19.220+0000 [id=25] INFO h.plugins.ec2.EC2AbstractSlave#stop: EC2 instance stop request sent for same-instance-id
2020-12-28 12:38:01.182+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Launching instance: same-instance-id
2020-12-28 12:38:01.183+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: bootstrap()
2020-12-28 12:38:01.184+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Getting keypair...
2020-12-28 12:38:01.184+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Using private key jenkins (SHA-1 fingerprint same-fingerprint)
2020-12-28 12:38:01.184+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Authenticating as jenkins
2020-12-28 12:38:01.286+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to null on port 22, with timeout 10000.
2020-12-28 12:38:01.287+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to null:22
2020-12-28 12:38:01.288+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
2020-12-28 12:38:06.363+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to null on port 22, with timeout 10000.
2020-12-28 12:38:06.364+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to null:22
2020-12-28 12:38:06.364+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
On other jenkins server launch says:
Dec 28, 2020 7:15:36 AM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to private-ip-ec2-instance.compute.internal on port 22, with timeout 10000.
Dec 28, 2020 7:15:36 AM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: There was a problem while connecting to private-ip-ec2-instance:22
Dec 28, 2020 7:15:36 AM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
---
Originally reported by 
nlopyrev, imported from: EC2 node not start after stop/disconnect with parameter Idle termination time
Raw content of original issue
We have a problem - after stopping ec2, the nodes do not start again. Playable on three jenkins servers. One of them is clean with plugin Amazon EC2.
Plugin settings and access to ec2 are the same
Common settings on plugin EC2
Idle termination time: 30
Stop/Disconnect on Idle Timeout: yes
Minimum number of instances: 0
Minimum number of spare instances: 0
Host key Verification Strategy: OffConnection stratagy: PublicDNS
When I checked credentials and AMI - plugin says - success
Logs on one node when I launch
2020-12-28 12:33:18.949+0000 [id=25] INFO h.p.ec2.EC2RetentionStrategy#internalCheck: Idle timeout of EC2 (Test Amazon connection) - Jenkins PHP70 (same-instance-id) after 30 idle minutes, instance statusRUNNING
2020-12-28 12:33:18.950+0000 [id=25] INFO h.plugins.ec2.EC2AbstractSlave#idleTimeout: EC2 instance idle time expired: same-instance-id
2020-12-28 12:33:19.220+0000 [id=25] INFO h.plugins.ec2.EC2AbstractSlave#stop: EC2 instance stop request sent for same-instance-id
2020-12-28 12:38:01.182+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Launching instance: same-instance-id
2020-12-28 12:38:01.183+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: bootstrap()
2020-12-28 12:38:01.184+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Getting keypair...
2020-12-28 12:38:01.184+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Using private key jenkins (SHA-1 fingerprint same-fingerprint)
2020-12-28 12:38:01.184+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Authenticating as jenkins
2020-12-28 12:38:01.286+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to null on port 22, with timeout 10000.
2020-12-28 12:38:01.287+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to null:22
2020-12-28 12:38:01.288+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
2020-12-28 12:38:06.363+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Connecting to null on port 22, with timeout 10000.
2020-12-28 12:38:06.364+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Failed to connect via ssh: There was a problem while connecting to null:22
2020-12-28 12:38:06.364+0000 [id=1078] INFO hudson.plugins.ec2.EC2Cloud#log: Waiting for SSH to come up. Sleeping 5.
On other jenkins server launch says:
Dec 28, 2020 7:15:36 AM hudson.plugins.ec2.EC2Cloud
INFO: Connecting to private-ip-ec2-instance.compute.internal on port 22, with timeout 10000.
Dec 28, 2020 7:15:36 AM hudson.plugins.ec2.EC2Cloud
INFO: Failed to connect via ssh: There was a problem while connecting to private-ip-ec2-instance:22
Dec 28, 2020 7:15:36 AM hudson.plugins.ec2.EC2Cloud
INFO: Waiting for SSH to come up. Sleeping 5.
environment
```
Debian 10
Jenkins 2.263.1, 2.230
Plugin Amazon EC2 version 1.56, 1.53
```
1 attachment
- [CleanShot 2022-03-10 at 21.20.33@2x.png](https://issues.jenkins.io/secure/attachment/57439/CleanShot%202022-03-10%20at%2021.20.33%402x.png)
> 
Contributor guide
Research direction
Start by tracing the EC2RetentionStrategy, EC2AbstractSlave, and EC2Cloud log paths around idle timeout, stop, and bootstrap. Reproduce the stop-and-restart scenario with the reported Idle termination time and compare the null SSH host with the working private-DNS case. Done means the stopped EC2 node starts again and the plugin retries SSH using a valid host.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100