Heartbeats between master and worker processes.

Heartbeats between master and workers have been mentioned before. Seeing https://github.com/JuliaLang/julia-ipython/issues/8 triggered it again.

Suggesting a design :

• The requirement is to provide an alternate communication channel to a Julia worker process that can be used for 2 things presently - a) heartbeats and b) querying the state of the process (including long running computation progress)
• This has to be in a different thread since a long running compute bound task will not yield frequently enough
• The handler also needs to be in C since core Julia is not yet thread-safe
• Currently the accept_handler in multi.jl waits for connection requests from peer processes.
• Upon a new connection it will now read a a short 4-byte header field that will specify what type of connection it is.
• If the client has sent "WRKR", it will call create_message_handler_loop as it currently does
• If the client has sent "ADMN", it will disengage the socket from libuv, start a new OS thread and execute a C "admin" function in the new thread with this socket. Only one admin thread can be running at any time.
• The "admin" function will currently just respond to "ping" messages with some state information about the worker process. Maybe also provide a means for https://github.com/timholy/ProgressMeter.jl to publish compute progress information which is returned with every ping. Will use a pipe to push information between the Julia thread and the admin thread.
• If a ping message has not been received for a specified timeout (say 2 minutes) or this socket connection breaks, the worker exits if configured to do so.
• Julia master process too starts a new thread (C code) that continually sends these heartbeat messages to all connected workers and exposes the returned state information to the Julia thread. A show_workers() can pretty print this information.

The whole heartbeat mechanism can be switched on via a command line argument to the julia executable (default is off)

It seems workable enough and we don't need the admin thread to listen on a different port. Do let me know what you guys think.

Start by reading the accept_handler in multi.jl and the existing create_message_handler_loop path. Compare the proposed WRKR and ADMN connection handling, C admin thread, heartbeat messages, and state reporting against the current worker protocol. Before implementation, define the accepted protocol, configuration behavior, and tests that would demonstrate completed heartbeat and failure handling.