local_resource can leave running processes after Tilt is terminated

Expected Behavior

• All local_resource processes terminate when Tilt terminates
  • This is different than k8s_resource behavior which are left until tilt down but is the expected current behavior

Current Behavior

• Processes can be left behind under some circumstances
  • Tilt quits unexpectedly (e.g. panic due to a Tilt bug)
  • User hits Ctrl-C twice in rapid succession
  • Process doesn't respond to SIGTERM
    • Tilt only waits 2 seconds on shutdown, we only send a SIGKILL if process takes more than 30 secs to respond to SIGTERM

Steps to Reproduce

See cases above - a good real world example is docker run --rm ..., especially if it doesn't respond well to SIGTERM which is fairly common, especially with entrypoint trampolines in some containers

Context

There's no way to ask the OS to kill all child processes when the parent exits. The most robust way looks to be to use an intermediate process that keeps an eye on the parent process and then kills the real child and exits itself.

We can at least do a little bit better here in the exit code by ensuring we SIGKILL everything before giving up after 2 seconds and/or on second Ctrl-C.

Start by tracing how local_resource processes are started and how Tilt handles shutdown and repeated Ctrl-C. Reproduce the listed termination cases, then verify that all local_resource processes are forcibly terminated within the expected shutdown behavior, including when SIGTERM is ignored.