mudler / mudler/vllm.cpp

ROCm: release captured graphs when instantiation fails

Open
#3,049 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
C++
Stars
423
Forks
53
Avg merge
20h 26m
Merged PRs (30d)
310

Description

Row: BACKEND-ROCM

At e1948ecebde085907171abce41a7b2e7541fd7d8, RocmBackend::EndCaptureGraph obtains a raw hipGraph_t from hipStreamEndCapture, then checks hipGraphInstantiate before destroying that raw graph (src/vt/rocm/rocm_backend.hip:360–377). An instantiate error throws past hipGraphDestroy, so the captured raw graph loses its cleanup path. The single-slot EndCapture path has the same ordering at lines 342–351.

This is a source-confirmed cleanup gap. Controlled hardware error-injection evidence is still pending. The graph seam originated in 7ba9a675f (#473). The open-issue search for hipGraphInstantiate returned #332, whose broader historical activation issue does not track this failure cleanup.

Fix this in the current #3041 graph flow under its committed .agents/specs/rocm-gfx1100-static-graph.md capture-failure and partial-construction obligations. Preserve the primary instantiate error, destroy owned raw handles exactly once, and retain successful capture/replay behavior. Cover both affected entry points and any partial executable ownership without changing provider, dtype, sampling, graph-default or dedup policy.

Before the repair, run a small immutable test against the unchanged baseline. It must enter the real backend, capture a real HIP graph, inject one instantiate failure, and observe raw-handle cleanup. Preserve the intended red, focused green, effective deletion mutation, fresh review and operator rerun. Root holds the local GPU mutex. The landing PR must reference this issue and close it when the repair lands; filing it does not defer the fix.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in src/vt/rocm/rocm_backend.hip at RocmBackend::EndCaptureGraph and the single-slot EndCapture path, then read the committed .agents/specs/rocm-gfx1100-static-graph.md obligations. Run the immutable baseline test against a real HIP graph with one injected instantiate failure. Done means both entry points preserve the primary error, release owned raw handles exactly once, and retain successful capture/replay behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.