Avoiding conflicts between servers through a DSC configuration Database

One of the issues we've had from day 1 of SharePointDsc has been how to avoid conflicts between resources running on different servers. Right now we have an "after the fact" approach that involves catching that error messages from SharePoint that talk about "update conflicts" and then flagging to the DSC engine that a reboot is pending as a way to loop back around and trying again, but this isn't great. It also won't help us for things like two servers trying to create a farm at the same time if we merge SPCreateFarm and SPJoinFarm as discussed in #455. We need a new approach here.

After spitballing some ideas for this around with a few of my colleagues the best idea we came up with was to use a database to manage "locks". Here's a run down of how the whole process would work:

1. A new resources (called SPDscConfigDatabase or something) would be used to provision a configuration database - it would take a property name of the server that would be responsible for provisioning the resource out the database
2. This resource would create registry entries to store the connection string and details about the config DB here
3. When a resource that would require a lock runs, it would check for the registry entries created by the resource above and then attempt to create an entry in a table where a primary key constraint would ensure that no two servers could create the same entry for a lock at the same time, so we now can make sure that things only happen on one server at a time
4. After the resource finishes (pass or fail) it removes the lock entry from the DB

This could help us solve things like the SPFarm resource, as well as then giving us some options for running resources that would typically just run on one server (like SPWebApplication) on multiple servers which can improve the high availability of configuration controlled through SharePointDsc.

To help with backwards compatability as well we could also make sure that if the registry entires aren't set by the new config DB resource then we skip that lock check and just do it. We can also wrap a lot of this sort of thing up in the helper methods to make sure we minimise the amount of change we need to make to other parts of the code throughout all the resources as well.

Now this is definitely a big enough change to make me want to include it in the v2.0 discussions (issue #456) rather than try to make it fit the current model. But I wanted to throw it out there for some discussion to see what people think of the approach.

Start by reading the proposed v2 context in issues #455 and #456, then review this issue's SPDscConfigDatabase, registry-entry, and lock-table proposal. Done would require an agreed design for provisioning the database, coordinating locks, preserving the no-database fallback, and integrating the approach across affected SharePointDsc resources.