compiler-explorer / compiler-explorer/infra
ce puts instance in standby even if no healthy instances remain
- Dominant language
- Python
- Stars
- 434
- Forks
- 429
- Avg merge
- 4d 21h
- Merged PRs (30d)
- 38
Description
If there's just one instance around in (say) live, then things go wrong:
```
ubuntu@ip-172-30-0-97 ~> ce --env prod restart
Running in prod
Confirm restart version Release(1982, master, dist/travis/master/1982.tar.xz, 40276304, 3851d6..0247b4) in env prod
Type the name of the environment: prod
INFO:ce:Restarting i-0d080e6b96663f5ba@54.172.93.75...
INFO:ce:Enabling instance protection for i-0d080e6b96663f5ba@54.172.93.75
INFO:ce:Group 'Gcc Explorer' needs to be adjusted to keep enough nodes
INFO:ce:Putting i-0d080e6b96663f5ba@54.172.93.75 into standby
INFO:ce:Waiting for i-0d080e6b96663f5ba@54.172.93.75 to reach autoscale lifecycle 'Standby'...
^TINFO:ce:...done
INFO:ce:Restarting service on i-0d080e6b96663f5ba@54.172.93.75
INFO:ce:Moving i-0d080e6b96663f5ba@54.172.93.75 out of standby
INFO:ce:Waiting for i-0d080e6b96663f5ba@54.172.93.75 to reach autoscale lifecycle 'InService'...
INFO:ce:...done
INFO:ce:Waiting for i-0d080e6b96663f5ba@54.172.93.75 to reach ELB state 'healthy'...
ssh: connect to host 54.172.93.75 port 22: Connection refused
WARNING:ssh:Failed to execute on remote host: Command '['ssh', '-o', 'ConnectTimeout=5', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', 'ubuntu@54.172.93.75', '--', "'sudo'", "'systemctl'", "'show'", "'compiler-explorer'"]' returned non-zero exit status 255
ERROR:ce:Failed restarting i-0d080e6b96663f5ba@54.172.93.75 - skipping: Instance no longer running (state shutting-down)
INFO:ce:Putting desired instances for Gcc Explorer back to 1
```
Main thing is putting the one good instance into standby _before_ the node count reaches two healthy.
Contributor guide
Assessment
This issue has not been assessed yet.