ORNL / ORNL/ReSolve

rework workspaces

Open
#345 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cuda enhancement hip question
Dominant language
C++
Stars
84
Forks
13
PR merge metrics
No merged PRs in 30d

Description

currently, the way we handle workspaces for matrix operations (which are stored in the resolve/workspace directory) is flawed.

while #343 provided a stopgap solution, we still bundle together the handles to the device (if there is one) with the auxiliary memory used for matrix operations. this is a problem because there are operations such as the transpose operations which require sizing the buffer to the sparsity pattern of the input matrix and changing the sparsity pattern of the input matrix then requires the user to delete the old transpose buffer and make a new one. if a user of resolve wanted to transpose several matrices where many of them have the same sparsity pattern, this requires them to order their work beforehand (if this is even possible), maintain several workspaces, or recreate the transpose buffer several times. while we could make this easier by maintaining several transpose buffers at a time, this puts more work on our end to do this, and doing that would also add more overhead. i instead propose the following:

  • we create a "memory handle" structure that holds memory for an operation on an unspecified device and functions like unique_ptr in that it deallocates itself once the scope is terminated
  • we create a separate "device handle" structure or subsume them into the matrix handler (the former is likely better as we may want handles elsewhere). these could also be freed once the scope is left
  • anything else is removed from the concerns of a generic structure

an example of how this could work is the following:

// A, At, B, Bt, C, Ct are matrices, where A and B are known to have the same sparsity pattern and A, B, C have matrices in them
// memorySpace and matrixHandler are already in scope

// create a device handle for our memory space
auto deviceHandle = DeviceHandle(memorySpace);

// create a memory handle for the transpose operation over A, B
auto memoryHandleAB = MemoryHandle::transpose(A, memorySpace, deviceHandle);

// create a memory handle for the transpose operation over C
auto memoryHandleC = MemoryHandle::transpose(C, memorySpace, deviceHandle);

// use them
matrixHandler.transpose(A, At, memoryHandleAB, deviceHandle);
matrixHandler.transpose(B, Bt, memoryHandleAB, deviceHandle);
matrixHandler.transpose(C, Ct, memoryHandleC, deviceHandle);

// memoryHandleAB, memoryHandleC, and deviceHandle are freed once this scope terminates

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in the resolve/workspace directory and review the stopgap from #343, then trace how workspace handles reach the matrix handler and transpose operations. Compare the proposed separate memory and device handles with the existing ownership and cleanup paths; done means transpose buffers can be shared by matching sparsity patterns while device and operation memory have independent lifetimes.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
hpc
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.