microsoft / microsoft/openvmm

nvme_driver: restored driver hangs on shutdown due to stale admin queue

Open
#3,219 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

storage testing
Dominant language
Rust
Stars
1.9k
Forks
238
Avg merge
1d 15h
Merged PRs (30d)
100

Description

After NvmeDriver::restore() in a unit test, calling shutdown() on the restored driver hangs forever.

Root cause

The admin queue always has a pending AER (Asynchronous Event Request) command. During restore, DMA blocks are freshly allocated at different physical addresses than the originals. The NVMe controller's internal SQ/CQ BAR addresses still point to the old blocks, so the AER completion never arrives. shutdown() waits for all pending commands to drain, which never happens.

What was tried
  • std::mem::forget(driver) — leaks but avoids the hang; not a real solution
  • update_servicing_flags(false) + shutdown() — triggers a controller reset, but the reset path itself waits for pending admin commands
  • Clearing nvme_keepalive and calling shutdown() — same hang
Possible fixes
  • Make the AER task cancelable so shutdown can kill it without waiting for a completion
  • Add a test-only shutdown_force() that drops queues without draining
  • Fix the DMA identity problem so the controller sees completions (depends on the PagePool blocker in #489)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the unit test that calls NvmeDriver::restore() followed by shutdown(), then trace the restored driver's admin queue and pending AER handling. Read the restore and shutdown entry points and consider the PagePool blocker in #489; done means shutdown completes without waiting forever for a stale AER completion.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.