galaxyproject / galaxyproject/gravity
Health check all instances before graceful
- Dominant language
- Python
- Stars
- 15
- Forks
- 20
- Avg merge
- 5h 33m
- Merged PRs (30d)
- 1
Description
Currently each instance is only checked before its turn to be restarted, meaning that if you have 2 gunicorns configured and instance 0 is up but instance 1 is down, doing a graceful will restart instance 0 (taking down all running gunicorns) and then once that's up, fail on the health check for instance 1.
We should instead health check all instances first and bail for admin intervention if they are not all healthy.
Alternatively, we could (perhaps optionally with a flag) order instances such that any stopped ones are first started and then running instances are restarted only after stopped instances are healthy.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the graceful restart flow and the per-instance health-check logic described in the issue. Verify how a stopped instance affects the restart sequence, then make the health-check behavior cover all instances before any running instance is taken down; done means graceful restart aborts safely when any instance is unhealthy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- devops, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100