Ensure api_fqdn Is Resolvable Within Milliseconds or log

ZenDesk 2718, Zendesk 2393

As a customer and employee of Chef, I would like Chef Server and Enterprise Chef server to warn me when name resolution is compromised with delay or failure.

1. Basic infrastructure like DNS should be reliable and performant, yes.
2. Chef Server should behave gracefully in the event of DNS misbehavior, or at the very least warn of the problem.

Currently, EC11.2.6 and presumably later versions are unable to commit sandboxes in the presence of multi second delays in node['api_fqdn'] resolution. The following is the only indication of the problem.

      2015-01-29_21:28:28.69569 [error] Checking presence of checksum: <<"59b2fe13f9c6a776b9f69eb00ac2b49f">> for org <<"c8d751a6b6d1445aa5cdc6c7552e4dee">>  from bucket "bookshelf" has taken longer than 5000 ms

      2015-01-29_21:28:28.72001 [error] {<<"method=PUT; path=/organizations/pedant-testorg-12311/sandboxes/c6c7552e4deef662ca00efba97d6f846; status=500; ">>, {error,{throw,{checksum_check_error,1},[{chef_wm_named_sandbox,validate_checksums_uploaded,2,[{file,"src/chef_wm_named_sandbox.erl"},{line,144}]}, {chef_wm_named_sandbox,from_json,2,[{file,"src/chef_wm_named_sandbox.erl"},{line,99}]},{webmachine_resource,resource_call,3,[{file,"src/webmachine_resource.erl"},{line,186}]}, {webmachine_resource,do,3,[{file,"src/webmachine_resource.erl"},{line,142}]},{webmachine_decision_core,resource_call,1,[{file,"src/webmachine_decision_core.erl"},{line,48}]}, {webmachine_decision_core,accept_helper,1,[{file,"src/webmachine_decision_core.erl"},{line,612}]},{webmachine_decision_core,decision,1,[{file,"src/webmachine_decision_core.erl"}, {line,517}]},{webmachine_decision_core,handle_request,2,[{file,"src/webmachine_decision_core.erl"},{line,33}]}]}}}

We should add monitoring of the time required for a gethostbyname or getaddress or whatever forces a name resolution. If it creeps above a default of 1000ms, we should begin to warn periodically in the erchef logfile.

The above would be much easier to diagnose and workaround if the erchef log contained something like this

2015-01-29_21:28:28.72001 [error] {<<"Heyo, I am seeing an average of 1800s delays for resolving chef-server1.something.local">>, {error,{throw,{name_resolution_check_error,1}

Start with src/chef_wm_named_sandbox.erl at validate_checksums_uploaded/2 and trace the api_fqdn name-resolution path. Determine how to measure and periodically log delays above the proposed 1000 ms threshold, then verify sandbox commits produce an actionable erchef log warning.