FP16_Optimizer incompatible with ordinary PyTorch optimizers (non-FusedAdam)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9k
- Forks
- 1.5k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 3
Description
https://github.com/NVIDIA/apex/commit/be42aad5ec57440dae478616b1054046dcefa2df#diff-28573bfd254eee1f6d12ad87a20007d8R143 broke compatibility between FP16_Optimizer and PyTorch's built-in optimizers by introducing keyword arguments in its call to optimizer.step that don't exist on the built-in optimizers. This causes exceptions like
File "/home/hsw-server-ubuntu/anaconda3/envs/pytorch/lib/python3.6/site-packages/apex-0.1-py3.6-linux-x86_64.egg/apex/optimizers/fp16_optimizer.py", line 160, in step
grad_norms=norm_groups)
TypeError: step() got an unexpected keyword argument 'grads'
which is from the original (closed) issue #129. This appears to be accidental since the documentation has an example using FP16_Optimizer with SGD: https://github.com/NVIDIA/apex/blob/47e3367fcd6636db6cd549bbb385a6e06a3861d0/apex/optimizers/fp16_optimizer.py#L243-L244
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in apex/optimizers/fp16_optimizer.py around line 160 and compare the optimizer.step call with the documented SGD example. Verify how ordinary PyTorch optimizers handle the call, then confirm that the documented FP16_Optimizer-with-SGD usage no longer raises the reported unexpected-keyword exception.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100