compiler-explorer / compiler-explorer/infra

ce puts instance in standby even if no healthy instances remain

Open
#54 0 comments 0 reactions 1 assignee Claimed by @mattgodbolt View on GitHub
Dominant language
Python
Stars
434
Forks
429
Avg merge
4d 21h
Merged PRs (30d)
38

Description

If there's just one instance around in (say) live, then things go wrong:

```
ubuntu@ip-172-30-0-97 ~> ce --env prod restart
Running in prod
Confirm restart version Release(1982, master, dist/travis/master/1982.tar.xz, 40276304, 3851d6..0247b4) in env prod
Type the name of the environment: prod
INFO:ce:Restarting i-0d080e6b96663f5ba@54.172.93.75...
INFO:ce:Enabling instance protection for i-0d080e6b96663f5ba@54.172.93.75
INFO:ce:Group 'Gcc Explorer' needs to be adjusted to keep enough nodes
INFO:ce:Putting i-0d080e6b96663f5ba@54.172.93.75 into standby
INFO:ce:Waiting for i-0d080e6b96663f5ba@54.172.93.75 to reach autoscale lifecycle 'Standby'...
^TINFO:ce:...done
INFO:ce:Restarting service on i-0d080e6b96663f5ba@54.172.93.75
INFO:ce:Moving i-0d080e6b96663f5ba@54.172.93.75 out of standby
INFO:ce:Waiting for i-0d080e6b96663f5ba@54.172.93.75 to reach autoscale lifecycle 'InService'...
INFO:ce:...done
INFO:ce:Waiting for i-0d080e6b96663f5ba@54.172.93.75 to reach ELB state 'healthy'...
ssh: connect to host 54.172.93.75 port 22: Connection refused
WARNING:ssh:Failed to execute on remote host: Command '['ssh', '-o', 'ConnectTimeout=5', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', 'ubuntu@54.172.93.75', '--', "'sudo'", "'systemctl'", "'show'", "'compiler-explorer'"]' returned non-zero exit status 255
ERROR:ce:Failed restarting i-0d080e6b96663f5ba@54.172.93.75 - skipping: Instance no longer running (state shutting-down)
INFO:ce:Putting desired instances for Gcc Explorer back to 1
```

Main thing is putting the one good instance into standby _before_ the node count reaches two healthy.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.