llnl / llnl/LEAP

cuda memory leak

Open
#170 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Cuda
Stars
249
Forks
37
PR merge metrics
No merged PRs in 30d

Description

**when this operator is called repeatedly during iterations, the GPU memory usage keeps increasing, finally the error "cuda 3D Allocation error:out of memory "occurs. And this phenomenon only occurs in ubuntu system, not in window system **

class projector_t(nn.Module):
def __init__(self):
super(projector_t, self).__init__()

def forward(self, sino, index, batchSize, d_angle=0):
device_name = "cuda:1"
device = torch.device(device_name)
proj = Projector(forward_project=False, use_static=True, use_gpu=True, gpu_device=device, batch_size=batchSize)
# proj = FBP(forward_FBP=True, use_static=True, use_gpu=True, gpu_device=device, batch_size=batchSize)
numCols = 1024
numTurns = 1024 / 1024
numAngles = int(1024 * numTurns)
pixelHeight = 1.12
pixelWidth = 1.12
# Set the number of detector rows
numRows = 32
phis = proj.leapct.setAngleArray(numAngles, 360.0 * numTurns)
# Set the scanner geometry
proj.leapct.set_conebeam(numAngles, numRows, numCols, pixelHeight, pixelWidth, 0.5 * (numRows - 1),
0.5 * (numCols - 1) - 4.4912, phis+d_angle, 570, 1005, tau=0.0)
proj.leapct.set_curvedDetector()
# Set the helical pitch.
proj.leapct.set_normalizedHelicalPitch(1.6)
pixels = 512
slices = 5
voxelWidth = 0.6
voxelHeight = 1.0
proj.leapct.set_volume(pixels, pixels, slices, voxelWidth, voxelHeight)
proj.allocate_batch_data()
g = sino
f = proj.leapct.allocateVolume()
f = torch.from_numpy(f).to(device)
# proj.leapct.set_FORBILD(f, True)
f[:] = 0.0
# Reconstruct the data
proj.leapct.FBP(g, f)
# f = proj(g)
del sino, g, proj
return f

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the shown projector_t.forward path and its repeated Projector construction, then measure GPU memory across repeated calls on Ubuntu and Windows. Done means repeated iterations no longer steadily increase GPU usage or end in a CUDA 3D allocation out-of-memory error.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.