Progress information flows slowly

Not that slowly, not to worry. ;)

Each round of execution, a Subgraph will do one round of exchange of progress information with other workers. This is sane, and I don't know that we should do it any differently. However, it has the defect that for a single-threaded computation, a subgraph with a sequence of n operators that hold capabilities until their frontiers advance will require n rounds of execution of the hosting Subgraph before moving data through each operator. In each round, progress information makes it to the next operator, but that operator's progress is not reflected until the next round of execution.

Alternately, we could flow worker-local progress information forward after each operator, and accumulate the information for exchange with other workers. This could accelerate single-threaded computations, and perhaps computations where one worker is holding up the show (although the edges to operators requiring progress information are probably exchange edges).

There is the additional limitation that an operator is only informed of progress on its inputs once it has consumed them, which means returning control back to the Subgraph for each operator. Perhaps the progress information supplied to each operator could provide enough information for the operator to locally update the information (e.g. in response to message counts), or the progress information could be indirectly hosted by something that is updated as the messages are drained (currently, the operator itself reports the count back to the parent Subgraph, who does some math and next-time-around informs the operator that all messages are consumed).

Another option is to more aggressively incrementalize the progress flow inside a Subgraph, so that we only encounter and execute child operators who have experienced change in their frontiers. This is probably tricky, in that we should still be scheduling these operators so they can read from their inputs and such, but when there are n-1 operators with nothing to do it feels silly to check with each of them to confirm this, over and over.

Start by tracing the Subgraph execution loop and its worker-local progress exchange. Compare the proposed approaches for forwarding progress, updating it while messages are drained, or scheduling only operators with changed frontiers. Done would require a decided design and evidence that progress moves faster without breaking operator scheduling.