NVIDIA-NeMo / NVIDIA-NeMo/RL

transformers 5.3.0 requires accelerate >= 1.1.0

Open
#2,221 0 comments 0 reactions 1 assignee Claimed by @terrykong View on GitHub
bug community-request waiting-on-maintainers
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

NemoRL `pyproject.toml` has `transformers==5.3.0` and `accelerate>=0.26`. However this version of transformers requires `accelerate>=1.1.0`.

Caught this crash because of it:

```
(XXXWorker pid=1783492) File "/opt/ray_venvs/nemo_rl.models.policy.workers.XXX/lib/python3.12/site-packages/accelerate/big_modeling.py", line 135, in register_empty_parameter [repeated 6x across cluster]
(XXXWorker pid=1783492) module._parameters[name] = param_cls(module._parameters[name].to(device), **kwargs) [repeated 6x across cluster]
(XXXWorker pid=1783492) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [repeated 6x across cluster]
(XXXWorker pid=1783492) TypeError: Parameter.__new__() got an unexpected keyword argument '_is_hf_initialized' [repeated 6x across cluster]
(XXXWorker pid=1783492) [rank6]:[W406 20:06:50.993913153 ProcessGroupNCCL.cpp:1553] Warning: WARNING: destroy_process_group() was not called before program exit, which can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) [repeated 6x across cluster]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.