jenkinsci / jenkinsci/openstack-cloud-plugin
RFE: Avoid using broken templates/clouds
- Dominant language
- Java
- Stars
- 49
- Forks
- 97
- Avg merge
- 13h 23m
- Merged PRs (30d)
- 6
Description
I've found that OpenStack systems crash and burn - sometimes they just stop accepting connections, sometimes they accept connections but never answer, sometimes they answer but always fail.
I've also found that it's easy for an admin to enter in a template that doesn't work and always fails to spin up, e.g. the wrong network id.
In both cases, the openstack-cloud plugin will blindly ignore the failures and continue to try to spin up instances, and will continue to tell Jenkins "yes, I can do this" (i.e. `JCloudsCloud.canProvision(Label)` will return true) even when it's doomed to failure, and this means that Jenkins doesn't ask any other cloud to try, so one failing cloud/template can take out your entire Jenkins server.
We need to fix that - one failing template or cloud shouldn't stop the other templates/clouds from being able to shoulder the load.
Proposal:
- Add a back-off period (in seconds) to the cloud configuration. Default it to a minute or two.
- If a template fails, we record the failure and the failure time in a transient field (so it isn't persisted) in the template.
- When `JCloudsCloud.canProvision(Label)` is called, it takes into account the time of the last failure and excludes any templates that failed too recently. This may well mean that we answer "no" when we would otherwise say "yes".
- When `JCloudsCloud.provision(...)` is called, we sort our templates in order of last failure so that templates that have never failed come first and the most recently failed templates are considered last. That way we'll allow all templates to be tried in sequence until we find one that works or end up ruling them all out.
- More serious issues like timeouts, a failure to log in etc, should be recorded against the cloud in similar (transient) fields and should rule out the entire cloud for a period.
- The configuration UI should show these issues, allowing the administrator to see the last exception(s) that occurred and when they happened, to guide them in how to fix things. e.g. if a template can't work because we've got an invalid network-id, that should show up in the UI for the template. If a OpenStack cloud locks up and stops responding, causing timeouts when talking to it, that should show up in the UI for the cloud as a whole.
Note: I added functionality that did this to the docker-plugin in its [issue#626](https://github.com/jenkinsci/docker-plugin/issues/626) so my Jenkins server could run properly even when there were problems with some of the docker hosts it was using; I'm now having problems with some of my OpenStack clouds...
Contributor guide
Assessment
This issue has not been assessed yet.