Shipit Revision check improvements

Context

Shipit is pretty much oblivious to what is running in production.
It was built on the idea that a revision is deployed or not.
When a deploy is triggered, if the deploy script crash, Shipit consider that nothing changed on our production servers.

The only feature we had, was to poll every minutes for the production revision, so that Shipit could be aware of manual deploys and reflect theses in it's data model.

But even this is based on the idea that deploys are atomic, and that a single revision is running at any moment.

Unfortunately this isn't true.

Early work

https://github.com/Shopify/shipit-engine/pull/489 introduce a flapping status in Shipit, to reflect the situations were we're running 2 revisions.

The status itself make sense, but the way to detect it doesn't really. It reacts to inconsistent responses from the script. e.g:

• Deploy fail
• Revision check says the new revision is running
• Once the the check return the same result 5 times in a row, it is considered as the new truth.

This is a bit silly, the script should be able to tell shipit that 2 revisions are running immediately.

Use cases

Manual deploy / rollback

This use case should still be supported. If someone deployed the stack manually, Shipit should be able to detect it to prevent any confusion.

Failed non-transactionnal deploy

The deploy script crash leaving nodes on 2 different revisions. Shipit need to mark the deploy as flapping until the situation is resolved.
Once we're back on a single revision, Shipit should mark this deploy as either success or failure based on which revision is finally in production.

Proposal

We could have the script return a specific error code if multiple revisions are running:

• 0: A single revision is running. That revision is printed on STDOUT.
• 42: Multiple revisions are running. No specific output is expected.

Non-zero exits other than 42 are ignored. This is so that application with transactional deploys (e.g. heroku apps) can stay with the old, simple solution.

Shipit would then immediately mark the last deploy as flapping.

This would be the only way a deploy can be marked as flapping, meaning we remove the "5 times in a row" code.

@eapache @csfrancis @Sirupsen thoughts?

Start by tracing Shipit's revision-check and deploy-status handling, then review the deploy script exit-code contract described in the issue. Define how code 0, code 42, and other non-zero exits affect manual deploys, failed deploys, and flapping status. Done means the behavior is specified and validated for single-revision, multi-revision, and recovery cases.