apex mixed precision performance comparisons
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
I've done a lot of performance comparisons between apex (python-build version) and myself-implemented mixed precision code on ResNet, PreActResNet, ResNeXt and DenseNet with CIFAR-10. I found a few things really puzzling me:
- It seems myself-implemented mixed percision code actually works better in speed up training performance than apex. I simply follow the guide in quick start and added a few lines into my code. Did I miss something?
- For densenet, it seems
opt_level=O1performance worse as it took more time to complete training. Why this would happend?
Here is my comparison results. All the experiments are conducted in a single V100 and the package versions are the same for both of them.
For apex:
| Model | Opt Level (O0 / O1 / O2 / O3) | ||
|---|---|---|---|
| Acc | Time | GPU | |
| ResNet152 | 93.42 / 93.55 / 93,50 / 93.21 | 6h54min / 5h26min / 4h31min / 3h52min | 12855MiB / 7499MiB / 7425MiB / 6915MiB |
| PreActResNet152 | 93.27 / 93.39 / 93.34 / 93.65 | 6h44min / 5h26min / 4h24min / 3h47min | 12593MiB / 7323MiB / 7079MiB / 6783MiB |
| ResNext29_32x64d | 94.04 / 93.84 / 94.01 / 93.63 | 2h46min / 2h29min / 2h15min / 2h04min | 6348MiB / 3379MiB / 3673MiB / 3647MiB |
| DenseNet121 | 93.38 / 93.79 / 93.75 / 93.54 | 3h13min / 3h57min / 2h56min / 2h16min | 8823MiB / 5133MiB / 5129MiB / 5085MiB |
For implemented mixed precision code:
| Model | Standard | Mixed Precision(o scale) | Mixed Precision(w scale) | ||||||
|---|---|---|---|---|---|---|---|---|---|
| Acc | Time | GPU | Acc | Time | GPU | Acc | Time | GPU | |
| ResNet152 | 93.42 | 6h54min | 12855MiB | 93.44 | 3h49min | 7609MiB | 93.40 | 3h37min | 7601MiB |
| PreActResNet152 | 93.27 | 6h44min | 12593MiB | 93.31 | 3h42min | 7441MiB | 93.42 | 3h35min | 7575MiB |
| ResNext29_32x64d | 94.04 | 2h46min | 6438MiB | 93.84 | 1h59min | 3781MiB | 93.70 | 1h59min | 3783MiB |
| DenseNet121 | 93.38 | 3h13min | 8823MiB | 93.78 | 2h18min | 5235MiB | 93.78 | 2h19min | 5241MiB |
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the reported comparisons from the quick-start mixed-precision setup on a single V100, using the listed ResNet, PreActResNet, ResNeXt and DenseNet models. Compare apex opt_level O0/O1/O2/O3 with the two custom scaling approaches, focusing on DenseNet O1 and the reported training times, memory use and accuracy. Done means identifying a supported explanation for the performance differences.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100