Sequence parallelism is not supported for NemotronHForCausalLM
Open
community-request
enhancement
external
t-pytdensor
waiting-on-customer
x-sarvam
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
Is there a way use sequence parallelism for NemotronHForCausalLM model?
I am getting:
```
File "/data/TTS/aayush/nemo-rl/nemo_rl/models/policy/dtensor_policy_worker.py", line 352, in __init__
self.model = _parallelize_model(
^^^^^^^^^^^^^^^^^^^
File "/data/TTS/aayush/nemo-rl/nemo_rl/models/dtensor/parallelize.py", line 529, in _parallelize_model
return _parallelize_nm5_h(
^^^^^^^^^^^^^^^^^^^
File "/data/TTS/aayush/nemo-rl/nemo_rl/models/dtensor/parallelize.py", line 415, in _parallelize_nm5_h
assert not sequence_parallel, (
^^^^^^^^^^^^^^^^^^^^^
AssertionError: Sequence parallelism is not supported for NemotronHForCausalLM
```
Contributor guide
Assessment
This issue has not been assessed yet.