Lightning-AI / Lightning-AI/pytorch-lightning

A100 GPU MIG feature support for trainer

Open
#10,529 9 comments 0 reactions 2 assignees View on GitHub

@Borda is already working on this.

Since Nov 7, 2022.

feature trainer: argument
Dominant language
Python
Stars
31.4k
Forks
3.8k
Avg merge
6d 7h
Merged PRs (30d)
6

Description

## 🚀 Feature

A100 GPU MIG feature support for trainer

### Motivation

Hi, could you update trainer class for MIG feature GPU?

For our LSF scheduler , -gpu "num=1:mig=1" will give you a single smallest possible slice (1/7th)
 -gpu "num=1:mig=7" gives you a single entire A100 GPU (80G memory)(equivalent to the previous "num=1")

What will be the number of GPU argument for trainer? The real number of GPU , or the slice number?
for example num=1:mig=4, should it be gpu =1 or 4?

I found that torch.cuda.device_count() returns 1 but I am not sure if I pass the num 1 to -gpu of trainer, will it use 10G memory or 40G memory?

How about num=2:mig=4?

Can we pass -1 to use all available gpu? How will pytorchlightning decide the number of gpus? Can we get a print information ?

### Pitch

It would be better to make -gpus obvious,
for example: -gpus and -migs or treat the total slice of MIG from one A100 as one gpu.

### Alternatives

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.