ROCm / ROCm/iris

get_device_context tensor goes stale if heap_bases change after init

Open
#467 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug iris
Dominant language
Python
Stars
202
Forks
47
Avg merge
6d 11h
Merged PRs (30d)
4

Description

Bug

get_device_context() builds a new torch.tensor from self.heap_bases.tolist() on every call (see #466). Once #466 is fixed by precomputing the tensor in __init__, the context tensor will hold a snapshot of heap_bases at construction time.

If heap_bases were to change after init (e.g., via refresh_peer_access() after a new shmem.allocate() or as_symmetric() call with a future allocator), the precomputed context tensor would contain stale base addresses. Kernels using DeviceContext would translate pointers using wrong bases, causing silent data corruption or hangs.

Today this is not a bug — both the torch and vmem allocators produce stable heap_bases after the first refresh_peer_access(). But it will become one if an allocator ever remaps peer VA ranges.

Fix

After precomputing self._device_context in __init__, add an in-place update in refresh_peer_access():

self._device_context[2:2+self.num_ranks] = self.heap_bases

No allocation, CUDAGraph safe, one line.

Component

iris/iris.py, iris/symmetric_heap.py

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in iris/iris.py and inspect how the device context is initialized and refreshed, then review the related heap behavior in iris/symmetric_heap.py. Confirm that refresh_peer_access keeps the context's heap-base values current after heap_bases changes, without introducing allocations or breaking CUDA Graph safety.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.