Smarter handling of READ_WAIT_TIME and select calls

Writing down the discussion had with @tpowell-progress and @jaymzh today

Describe the Enhancement:

The select timeout using READ_WAIT_TIME has high latency for quick shellouts. I made a benchmark script for getting numbers, and on this simple example it would lead one to think READ_WAIT_TIME should be closer to 0.001 instead of 0.01. Syscalls aren't free, though, so that may create new issues on longer-running processes.

Instead, we should adjust the time given to the select calls based on how long a particular shellout is taking via @execution_time, with a sensible ceiling for long-running processes. This gets the best of both worlds for fast shellouts (eg systemctl is-enabled) and long running processes (eg dnf downloading the world), and doesn't require end-users to fool around with the READ_WAIT_TIME constant.

Describe the Need:

Chef Client runs where there's a variety of expected latencies means one-size-fits-all doesn't really exist for timeouts.

Current Alternative

The first alternative is simply clobbering the READ_WAIT_TIME constant, but if there's a variety of calls over the course of the run, this can get ugly fast.

We could make the READ_WAIT_TIME an optional default, but that means that you A) know the option exists, B) have instrumented how long a particular shellout typically takes, and C) that the latency of said shellouts never change.

Can We Help You Implement This?:

I think I can manage this one ;-)

Start in lib/mixlib/shellout/unix.rb at the select timeout and read READ_WAIT_TIME in lib/mixlib/shellout.rb, then review how @execution_time is tracked. Use the linked benchmark to understand the latency concern. Done means select waits adapt to the shellout's elapsed time while retaining a sensible ceiling for long-running processes.