Lightning-AI / Lightning-AI/pytorch-lightning
A100 GPU MIG feature support for trainer
@Borda is already working on this.
Since Nov 7, 2022.
- Dominant language
- Python
- Stars
- 31.4k
- Forks
- 3.8k
- Avg merge
- 6d 7h
- Merged PRs (30d)
- 6
Description
## 🚀 Feature
A100 GPU MIG feature support for trainer
### Motivation
Hi, could you update trainer class for MIG feature GPU?
For our LSF scheduler , -gpu "num=1:mig=1" will give you a single smallest possible slice (1/7th)
-gpu "num=1:mig=7" gives you a single entire A100 GPU (80G memory)(equivalent to the previous "num=1")
What will be the number of GPU argument for trainer? The real number of GPU , or the slice number?
for example num=1:mig=4, should it be gpu =1 or 4?
I found that torch.cuda.device_count() returns 1 but I am not sure if I pass the num 1 to -gpu of trainer, will it use 10G memory or 40G memory?
How about num=2:mig=4?
Can we pass -1 to use all available gpu? How will pytorchlightning decide the number of gpus? Can we get a print information ?
### Pitch
It would be better to make -gpus obvious,
for example: -gpus and -migs or treat the total slice of MIG from one A100 as one gpu.
### Alternatives
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.