upstairs panic from chaotic downstairs coming and going

In trying to reproduce #837, I managed to trigger an upstairs panic:

Jul 21 21:33:12.533 INFO Live Repair in progress, try again, component: crucible-0cedae45-3d6e-4d90-b2cb-56f1a1a42a89
Jul 21 21:33:12.539 WARN [1] flow control end , component: crucible-0cedae45-3d6e-4d90-b2cb-56f1a1a42a89
thread 'tokio-runtime-worker' panicked at 'Bad  job received in process_ds_completion: DownstairsIO { ds_id: 24849, guest_id: 23850, work: ExtentFlushClose { dependencies: [24779, 24777, 24776, 24774, 24773, 24771, 24770, 24767, 24765, 24763, 24761, 24759, 24757, 24755, 24752, 24750, 24749, 24747, 24746, 24744, 24743, 24741, 24740, 24738, 24737, 24735, 24734, 24733, 24731, 24730, 24729, 24727, 24726, 24724, 24723, 24722, 24720, 24719, 24718, 24716, 24715, 24713, 24712, 24711, 24709, 24707, 24705, 24702, 24701, 24699, 24698, 24696, 24695, 24693, 24692, 24690, 24689, 24687, 24686, 24684, 24683, 24681, 24680, 24678, 24677, 24675, 24674, 24672, 24671, 24669, 24668, 24666, 24665, 24663, 24662, 24661, 24658, 24657, 24654, 24651, 24649, 24648, 24647, 24645, 24644, 24643, 24642, 24641, 24639, 24638, 24636, 24635, 24634, 24632, 24628, 24625, 24624, 24622, 24621, 24620, 24619, 24618, 24616, 24615, 24614, 24612, 24611], extent: 0, flush_number: 318, gen_number: 2, source_downstairs: 2, repair_downstairs: [0] }, state: {0: Done, 2: Done, 1: Done}, ack_status: Acked, replay: true, data: None, read_response_hashes: [], impacted_blocks: InclusiveRange(ImpactedAddr { extent_id: 0, block: 0 }, ImpactedAddr { extent_id: 0, block: 1048575 }) }', /home/jordan/.cargo/git/checkouts/crucible-f3b5bdecdc6486d6/179e68b/upstairs/src/lib.rs:4335:21
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
./server.sh: line 9:  6436 Abort                   RUST_LOG=debug pfexec $SERVER run $CONF $ADDR:$PORT

Unfortunately I don't know exactly the series of events that triggered this, but was experimenting with repeatedly killing off and restarting a downstairs.

The panic is reported at upstairs/src/lib.rs:4335; start there and use the logged ExtentFlushClose state to trace the completion path. Reproduce it by repeatedly killing and restarting downstairs while investigating #837, then verify that the same churn no longer causes an upstairs panic.