facebookresearch / facebookresearch/fairscale
AttributeError: 'FlatParameter' object has no attribute '_full_param_padded'
- Dominant language
- Python
- Stars
- 3.4k
- Forks
- 293
- PR merge metrics
- No merged PRs in 30d
Description
I use the nllb branch of fairseq for nllb training.Encountering a problem:
File "/media/walter/work/hui.li/fairscale/fairscale/nn/data_parallel/fully_sharded_data_parallel.py", line 2302, in _gather_optim_state
desired_buffer_size = non_shared_params[0]._full_param_padded.size()
AttributeError: 'FlatParameter' object has no attribute '_full_param_padded'
srun: error: localhost: task 0: Exited with exit code 1
Contributor guide
Research direction
Reproduce the NLLB training failure, then inspect fairscale/nn/data_parallel/fully_sharded_data_parallel.py at _gather_optim_state and the access to _full_param_padded. Done means optimizer-state gathering completes without the AttributeError; add or run a focused regression test if the repository provides one.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100