vmm: guest event persistence holds the global state lock and blocks the async executor

Open
#1,269 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
48/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Active
Tech stack
rust
Domain
backend

Research direction

Trace HostApi.Notify into App::vm_event_report and inspect safe_write, the AppState mutex, and VM removal lifecycle. Add regression coverage for executor responsiveness, per-VM write ordering, cancellation, and removal safety; done means persistence no longer blocks the async executor while holding the global state lock.

Written by the indexing model from the issue text.

Description

Problem

HostApi.Notify calls synchronous App::vm_event_report on an async worker. An instance.info event executes safe_write (including file and directory fsync) while holding the global AppState mutex. A shutdown.progress event reporting powering off also writes its started marker under that mutex.

Slow storage therefore delays unrelated VMs’ control-plane state operations as well as the async worker. This is a source-level finding, not a measured production latency claim.

Fix

Move persistence off the executor and outside the global state lock. Preserve per-VM write ordering and serialize with VM removal, including when the notification request is cancelled. Add regression tests for responsiveness and lifecycle safety.

The supervisor log fsync changes in #1263 do not cover this call path.

Dominant language
Rust
Stars
546
Forks
96
Avg merge
19h 22m
Merged PRs (30d)
109

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Dstack-TEE/dstack

All issues in Dstack-TEE/dstack

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.