Propolis server panicked after hitting some `????????` rust_panic call stack

@augustuswm found a VM that had a propolis server panic. Here are the propolis log lines at the time:

18:41:28.222Z INFO propolis-server (vcpu_tasks): vCPU released from hold
    vcpu = 0
18:41:28.222Z INFO propolis-server (vm_state_worker): publishing new instance state
    gen = 9
    migration = InstanceMigrateStatusResponse { migration_in: None, migration_out: None }
    state = Running
18:41:28.222Z INFO propolis-server (vm_state_worker): State worker handled event
    outcome = Continue
18:41:28.222Z INFO propolis-server (vcpu_tasks): vCPU released from hold
    vcpu = 1
18:41:28.269Z INFO propolis-server: request completed
    latency_us = 55
    local_addr = [fd00:1122:3344:116::1:a6]:12400
    method = GET
    remote_addr = [fd00:1122:3344:116::1]:52233
    req_id = 8c5001b9-e06d-4606-acae-3fe5ab9cfdb0
    response_code = 200
    uri = /instance/state-monitor
18:41:28.637Z INFO propolis-server (ramfb): ramfb change
    config = Config { addr: 3198525440, fourcc: 875713112, flags: 0, width: 800, height: 600, stride: 3200 }
    state = valid
18:41:28.637Z INFO propolis-server (ramfb): notifying
18:41:28.637Z INFO propolis-server (vnc-server): pixel format set to fourcc=0x34325258
thread 'vcpu-0' panicked at bin/propolis-server/src/lib/vm/mod.rs:386:9:
vCPU 0: Unhandled VM exit: InstEmul(InstEmul { inst_data: [175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175, 175], len: 15 })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
[ Aug 12 18:41:34 Stopping because all processes in service exited. ]
[ Aug 12 18:41:34 Executing stop method (:kill). ]

Both the core file and propolis log can be found in /staff/rack3/BRM42220077/2024-08-29 (system-illumos-propolis-server:default.log.1723488297 and core.oxz_propolis-server_cdcb2537-a5ee-4e9d-97d8-d8a86b57d2dd.propolis-server.23389.1723488093.

@leftwo got the stack from the core file:

 fffff5ffef15b6ea _lwp_kill () + a                                                                                   
 fffff5ffef0ec6b2 raise (6) + 22                                                                                     
 fffff5ffef0c5dc8 abort () + 58                                                                                      
 0000000002592cd9 ???????? ()                                                                                        
 0000000002592cc9 ???????? ()                                                                                        
 000000000257b97d rust_panic () + d                                                                                  
 000000000257b636 std::panicking::rust_panic_with_hook::h0c682abf0664e68a () + 296                                   
 000000000257b362 std::panicking::begin_panic_handler::{{closure}}::h0d97730e7ba5f708 () + a2                        
 0000000002578db9 ???????? ()                             
 000000000257b0a6 ???????? ()                                                                                        
 00000000025bf522 ???????? ()                                                                                        
 00000000011957c8 ???????? ()                                                                                        
 000000000110f5ea ???????? ()                                                                                        
 000000000111412e core::ops::function::FnOnce::call_once{{vtable.shim}}::h19c41ab30f52a8dc () + 8e                   
 0000000002583dcb std::sys::pal::unix::thread::Thread::new::thread_start::h9ef84bfafda8297d () + 1b                  
 fffff5ffef154307 _thrp_setup (fffff5ffeef35240) + 77                                                                
 fffff5ffef154650 _lwp_start ()

Start with the panic site at bin/propolis-server/src/lib/vm/mod.rs:386, then inspect the referenced core file and propolis log in /staff/rack3/BRM42220077/2024-08-29. Use the stack and the InstEmul exit details to determine why the vCPU reaches an unhandled VM exit; done means the cause is explained and the failure is addressed or diagnosable.