Standardized script for toolkit health detection

From @arlake228 on March 14, 2015 12:20

Original issue 922 created by arlake228 on 2014-06-11T13:26:27.000Z:

For large scale deployments it would be very useful if there was a standardized way included in toolkit distributions for checking the overall health of the toolkit instance. A lot of these are capable of being detected individually in various ways, so the goal here isn't some groundbreaking new way of doing it but rather providing a standardized and simple way of querying the whole health in a single go.

For a first pass some useful metrics might be:

• number of scheduled tests in the queue
• average wait time for test availability in scheduler
• whether a given service such as bwctl or the MA is running
• whether the host is globally registered
• whether NTP is synced

For any of the metrics they should take options to dictate their thresholds and/or suppress checking them entirely.

Copied from original issue: perfsonar/project#920