depsolver concurrency limitations

(cross-filing as requested in opscode/chef-provisioning#112)

Using Open Source Chef Server 11.1.5 (but really ever since Chef 11) there seems to be a hard limit on the number of concurrent cookbook dependency resolutions.

We are running chef-client at regular intervals on all our hosts, but occasionally we want to kick them off immediately on one or more hosts, so we pdsh over them. This means that N (where N is the pdsh fanout) chef-client runs start at almost exactly the same time, and since they look up their cookbooks very early they nearly simultaneously hit the depsolvers.

Correlating with general chef-server load and the number of cookbooks / cookbook versions, it used to be that we could do this to no more than 5-10 nodes at the same time, in line with the default number of depsolvers (5). I bumped the erchef['depsolver_worker_count'] to 20 now and can do this to ~30 nodes at once without hitting errors, but see errors at 40. Since they're probably not perfectly in sync it looks to me like this is just hitting the new, higher limit again.

To sum up, it looks like each depsolver worker can only do one dependency resolution at a time, and it is not possible to have more than depsolver_worker_count simultaneous chef-client runs. Is that so, and is that by design or a known issue, or (quoting @jkeiser) "bad mojo"?

No source file or test is named. Start by reproducing concurrent cookbook dependency resolutions with different erchef['depsolver_worker_count'] values, then trace the depsolver worker entry point. Done means establishing whether the worker count limits simultaneous chef-client runs and recording the confirmed behavior or a well-scoped fix.