Chance of getting 409 while renewing the primary host lock

While investigating another issue I stumbled upon a lot of these 409s coming from renewing the primary host lock. I did some investigation and I believe that, because we do a 'fire and forget' with the host disposal, it's possible that a new host comes up and acquires the lock just before the old host releases it. That means that the next attempt to renew fails, as the lock has been released.

The error looks like: `Singleton lock renewal failed for blob 'locks/AppName/host' with error code 409: LeaseIdMismatchWithLeaseOperation. The last successful renewal completed at 2017-09-02T18:38:31.526Z (111006 milliseconds ago) with a duration of 57508 milliseconds. The lease period was 15000 milliseconds.`

I think it's happening with something like below.
*Note 1 -- this is all on a **single instance** and references to hosts are about the hosts being cycled via the `ScriptHostManager`.*
*Note 2 -- All of the host locks use the current machine id as the proposed lease, which allows multiple hosts on the same machine modify the lease.*

Host1 -> acquire lock
Host1 -> renew lock
*something triggers host recycle and creates a new host while disposing of the old host*
Host2 -> acquire lock
Host1 -> release lock
Host2 -> renew lock -> throw

It's actually possible for several other combinations to happen (I've seen a race where Host1 renews right as it releases, also causing a throw).

Ultimately, these errors seem benign as they'll eventually recover and all will be well, but it causes a lot of errors in the logs (I see 100k+ 409 logs that involve the host lock over the last week).

Start with the ScriptHostManager host-recycle sequence described in the issue, focusing on disposal, primary host-lock release, acquisition, and renewal ordering. Reproduce or trace the single-instance race and compare the resulting renewal logs with the reported 409 LeaseIdMismatchWithLeaseOperation errors. Done means the benign race is handled without generating the excessive error logs described.

Written by the indexing model from the issue text.