galaxyproject / galaxyproject/gravity

Health check all instances before graceful

Open
#163 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
15
Forks
20
Avg merge
5h 33m
Merged PRs (30d)
1

Description

Currently each instance is only checked before its turn to be restarted, meaning that if you have 2 gunicorns configured and instance 0 is up but instance 1 is down, doing a graceful will restart instance 0 (taking down all running gunicorns) and then once that's up, fail on the health check for instance 1.

We should instead health check all instances first and bail for admin intervention if they are not all healthy.

Alternatively, we could (perhaps optionally with a flag) order instances such that any stopped ones are first started and then running instances are restarted only after stopped instances are healthy.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the graceful restart flow and the per-instance health-check logic described in the issue. Verify how a stopped instance affects the restart sequence, then make the health-check behavior cover all instances before any running instance is taken down; done means graceful restart aborts safely when any instance is unhealthy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.