Maybe jobs should be able to block when they hit the max output size cap

In discussion with @plotnick: jobs currently are hard-killed if they exceed the (operator-defined) maximum stdout cap. This could be surprising, frustrating, or in some cases destructive, if the job is not, in some sense, "cancellation-safe", and the operator has misjudged (or forgotten to think about) how much output it will create.

Suggestion: it would be good if, instead of being kill -9'd, jobs which try to write past their quota block indefinitely, signaling back to the job manager that an "Output Too Big For He Gotdang File" event has occurred. By blocking the pipe into which the job writes, we apply backpressure to the job, which for most processes will stall them in some kind of reasonable way (because they, presumably, are meant to tolerate arbitrary I/O delays). This does not require SIGSTOP-ing the job, though we could do that too?

When such a pause occurs, we should set the status of the job to Paused, entering this back as a job-related event in the gossip network. Querying the status of the job, whether by polling or waiting for updates, should return to the client that the job has been paused. At this point, the client may make one of two decisions, the first of which is already wired in:

1. Actually kill the job (we already have this capability).
2. Allocate additional quota to the job as a per-job temporary override.

Either of these options, like all other job control commands, are broadcast back from the sush server into the gossip network, eventually reaching the sled which holds the paused job. If the client selects option 2, the job's max output size is increased, the concomitant backpressure is relieved, and it continues apace.

I don't think that much about this is difficult to wire up, because it operates through the same job control mechanisms that already exist. The most significant part, standing at a thousand feet away, is likely the backpressuring quota-enforcement pipe, which I suspect looks something like a triple of an AsyncRead, an AsyncWrite, and a watch::Receiver<usize>, which itself implements AsyncWrite, so that we can redirect output from the process into it.

Start by tracing the existing maximum-output quota enforcement and the job control path between the sush server, gossip network, and sled. Examine how the quota-enforcement pipe can apply backpressure and how paused status is reported. Done means an oversized-output job pauses, clients can observe and kill it or increase its temporary quota, and it resumes after the quota is raised.