dpd segfaulted in libdriver.sopipe_mgr_drv_completion_cb

Originally reported by @wfchandler:
During the R11 -> R12 upgrade for CUSTOMER we found core dumps from a number of services including sled-agent, pumpkind, and mgs, on sled14, the initial problem appears be have been due to a segfault in dpd. The same segfault occurred twice, the scrimlet was able to restore itself to normal operation.

Full cores are at /staff/CUSTOMER/mupdate-20250116-v12/cores/core.oxz_switch.dpd.1494.1731835599 and /staff/CUSTOMER/mupdate-20250116-v12/cores/core.oxz_switch.dpd.2454.1735771206. We failed to obtain the dendrite logs yesterday, but if necessary we could grab then during their next upgrade.

Quick summary of the dump:

> $C
fffff5ffb8c00210 libdriver.so`pipe_mgr_drv_completion_cb+0xadd()
fffff5ffb8c00280 libdriver.so`dr_process_completion+0xaf()
fffff5ffb8c00700 libdriver.so`lld_dr_service+0x12f()
fffff5ffb8c00730 libdriver.so`service_dr+0x43()
fffff5ffb8c00b60 libdriver.so`pipe_mgr_drv_service_ilist_drs+0x193()
fffff5ffb8c00fb0 libdriver.so`bf_switchd_process_async_dma_notifs+0x1b2()
fffff5ffb8c00fe0 libc.so.1`_thrp_setup+0x77(fffff5ffee8b1a40)
fffff5ffb8c00ff0 libc.so.1`_lwp_start()

> ::status
debugging core file of dpd (64-bit) from oxz_switch
initial argv: /opt/oxide/dendrite/bin/dpd run
threading model: native threads
status: process terminated by SIGSEGV (Segmentation Fault), addr=38

> fffff5ffb8c00210::regs -u
%rax = 0x2b75a775d9000007       %r8  = 0xfffff5ffdbf355af
%rbx = 0xfffff5ffb8c002c0       %r9  = 0xfffff5ffb8c00700
%rcx = 0x00001400da004156       %r10 = 0xfffff5ffdc89b3c0
%rdx = 0xfffff5ffb8c002d0       %r11 = 0x0000000000000098
%rsi = 0x0000000000000000       %r12 = 0x0000000000000024
%rdi = 0x0000000000000000       %r13 = 0x000000000000003a
                                %r14 = 0xfffff5ffe049c080
                                %r15 = 0xfffff5ffb8c00280

%cs = 0x3124e10 %fs = 0x0000    %gs = 0x0000
%ds = 0x46903b0e        %es = 0x46903b0e        %ss = 0x0000

%rip = 0xfffff5ffdc06ec7f libdriver.so`pipe_mgr_get_num_active_subdevices+0x3f
%rbp = 0x0000000000000000
%rsp = 0x0000000000000000

%rflags = 0x00136abd
  id=0 vip=1 vif=0 ac=0 vm=1 rf=1 nt=1 iopl=0x2
  status=<OF,df,IF,tf,SF,zf,AF,PF,CF>

%gsbase = 0x0000001c1cc9c219
%fsbase = 0x0000000000000000
%trapno = 0x3d28df
   %err = 0xb8c00700

> pipe_mgr_drv_completion_cb+0xadd::dis -b
fffff5ffdbfa3eb6  libdriver.so`pipe_mgr_drv_completion_cb+0xab6:movl   $0x4,%edi
fffff5ffdbfa3ebb  libdriver.so`pipe_mgr_drv_completion_cb+0xabb:popq   %r13
fffff5ffdbfa3ebd  libdriver.so`pipe_mgr_drv_completion_cb+0xabd:popq   %r14
fffff5ffdbfa3ebf  libdriver.so`pipe_mgr_drv_completion_cb+0xabf:popq   %r15
fffff5ffdbfa3ec1  libdriver.so`pipe_mgr_drv_completion_cb+0xac1:popq   %rbp
fffff5ffdbfa3ec2  libdriver.so`pipe_mgr_drv_completion_cb+0xac2:jmp    -0x181be7        <PLT=libtarget_sys.so`bf_sys_log_and_trace>
fffff5ffdbfa3ec7  libdriver.so`pipe_mgr_drv_completion_cb+0xac7:nopw   0x0(%rax,%rax)
fffff5ffdbfa3ed0  libdriver.so`pipe_mgr_drv_completion_cb+0xad0:movq   0x30(%r13),%rcx
fffff5ffdbfa3ed4  libdriver.so`pipe_mgr_drv_completion_cb+0xad4:testq  %rcx,%rcx
fffff5ffdbfa3ed7  libdriver.so`pipe_mgr_drv_completion_cb+0xad7:je     +0x5e9   <libdriver.so`pipe_mgr_drv_completion_cb+0x10c6>
fffff5ffdbfa3edd  libdriver.so`pipe_mgr_drv_completion_cb+0xadd:movq   %rax,0x38(%rcx) <<< SEGFAULT
fffff5ffdbfa3ee1  libdriver.so`pipe_mgr_drv_completion_cb+0xae1:movq   %rcx,(%rdx)
fffff5ffdbfa3ee4  libdriver.so`pipe_mgr_drv_completion_cb+0xae4:jmp    -0x595   <libdriver.so`pipe_mgr_drv_completion_cb+0x554>
fffff5ffdbfa3ee9  libdriver.so`pipe_mgr_drv_completion_cb+0xae9:movq   0xfffffffffffffba8(%rbp),%rcx
fffff5ffdbfa3ef0  libdriver.so`pipe_mgr_drv_completion_cb+0xaf0:movq   0xfffffffffffffb98(%rbp),%rbx
fffff5ffdbfa3ef7  libdriver.so`pipe_mgr_drv_completion_cb+0xaf7:addq   $0x1,%rcx
fffff5ffdbfa3efb  libdriver.so`pipe_mgr_drv_completion_cb+0xafb:movzbl 0x10(%rbx),%edx
fffff5ffdbfa3eff  libdriver.so`pipe_mgr_drv_completion_cb+0xaff:cmpl   %ecx,%edx
fffff5ffdbfa3f01  libdriver.so`pipe_mgr_drv_completion_cb+0xb01:jg     -0x6ab   <libdriver.so`pipe_mgr_drv_completion_cb+0x45c>
fffff5ffdbfa3f07  libdriver.so`pipe_mgr_drv_completion_cb+0xb07:movq   0x30(%r15),%rax
fffff5ffdbfa3f0b  libdriver.so`pipe_mgr_drv_completion_cb+0xb0b:testq  %rax,%rax

For more info, see: https://github.com/oxidecomputer/dendrite-archived/issues/1064

Begin with the two core dumps at /staff/CUSTOMER/mupdate-20250116-v12/cores/core.oxz_switch.dpd.1494.1731835599 and core.oxz_switch.dpd.2454.1735771206, then compare the reported stack, registers, and disassembly around pipe_mgr_drv_completion_cb. Done means identifying the segfault cause and demonstrating that dpd no longer crashes in this completion path.