SP serial console via MGS occasionally goes out to lunch

Start with the faux-mgs SP serial-console keepalive and retry behavior described in the report, including the UDP reconnect path and the logged communication failures. Reproduce the unresponsive-console condition if possible, then trace the SP-side task state and determine what change prevents the console from going out to lunch; done means the failure is understood and covered by a reliable verification procedure.

Anecdotally, operators using the faux-mgs <-> SP serial console will see the SP become unresponsive. In the past this resulted in a panic (https://github.com/oxidecomputer/management-gateway-service/issues/122); faux-mgs now automatically tries to "reconnect" (air quotes because this is all UDP; reconnect here means "the SP starts responding again"), which results in output like this that @sunshowers saw on dogfood today:

root@oxz_switch:~# Jan 18 22:25:06.876 WARN failed to send console keepalive, err: RPC call failed (gave up after 5 attempts), component: faux-mgs

logout

[Connection to zone 'oxz_switch' pts/1 closed]
BRM42220051 # ^C
BRM42220051 # Jan 18 22:25:31.985 WARN communication error with SP (will retry), err: RPC call failed (gave up after 5 attempts), component: faux-mgs
                                                                Jan 18 22:25:33.987 INFO communication with SP reestablished, component: faux-mgs

I haven't done any investigation into this. I just spot-checked BRM42220051 to check for a nonzero net task count (to check for evidence of the net watchdog kicking in), and the net count is still 0:

ID TASK                       GEN PRI STATE
 0 jefe                         ?   0 [cannot read supervisor memory]
 1 net                          0   5 recv, notif: eth-irq(irq61) wake-timer(T=2987784)
 2 sys                          0   1 recv
 3 spi2_driver                  0   3 recv
 4 i2c_driver                   0   3 recv
 5 spd                          0   2 notif: i2c1-irq(irq31/irq32)
 6 packrat                      0   1 recv
 7 thermal                      0   5 recv, notif: timer(T=2987652)
 8 power                        0   6 recv, notif: timer(T=2988561)
 9 hiffy                        0   5 notif: bit31(T=2987825)
10 gimlet_seq                   0   4 recv, notif: timer(T=2987713)
11 hash_driver                  0   2 recv
12 hf                           0   3 recv
13 update_server                0   3 recv
14 sensor                       0   4 recv, notif: timer(T=2988007)
15 host_sp_comms                0   7 recv, notif: jefe-state-change usart-irq(irq82) multitimer(T=2987701) control-plane-agent
16 udpecho                      0   6 notif: socket
17 udpbroadcast                 0   6 notif: bit31(T=2987746)
18 control_plane_agent          0   6 recv, notif: usart-irq(irq37) socket timer
19 sprot                        0   4 recv
20 validate                     0   5 recv
21 vpd                          0   4 recv
22 user_leds                    0   2 recv, notif: timer
23 dump_agent                   0   6 wait: reply from jefe/gen0
24 sbrmi                        0   4 recv
25 idle                         0   8 ready