Reduce memory usage in distributed triangle updates

Open
#5,942 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
5/5
Estimated time
Over a week
Newbie friendliness
25/100
Issue type
Feature
Clarity
Mostly clear
Activity status
Stale
Tech stack
cpp

Research direction

Start by reading tests/cpp/test_overlap.cpp around line 70 to understand the Swizzle1D reference, then review the distributed triangle update context from NVIDIA/Fuser#5890. The issue's proposed direction is to stream-parallelize allgather and reducescatter; done means reducing per-GPU memory to the stated O(b * s/dy * s/dx * c) bound.

Written by the indexing model from the issue text.

Description

Multi-GPU

This request comes from https://github.com/NVIDIA/Fuser/pull/5890. Currently, nvFuser uses too much memory by allgathering one of the einsum's operands.

Image

A better (not sure if the best) approach is to stream-parallelize the allgather and the reducescatter:

Image

This way, each GPU only has to store O(b * s/dy * s/dx * c).

In nvFuser, this can be represented as

Image

Note that

dy
|
s

is a Swizzle1D similar to https://github.com/NVIDIA/Fuser/blob/231d48c002629b6f8b33a9bae4ed9ff440d1777c/tests/cpp/test_overlap.cpp#L70

Dominant language
C++
Stars
404
Forks
84
PR merge metrics
No merged PRs in 30d

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from NVIDIA/Fuser

All issues in NVIDIA/Fuser

Similar issues

More C++ issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.