[QUESTION] Why FSDP2 does not support embedding tying
Open
question
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Your question**
```
AssertionError: --use-torch-fsdp2 requires --untie-embeddings-and-output-weights
```
Why does FSDP2 not support embedding tying? Torchtitan supports FSDP2 with embedding tying enabled at the same time.
Is it possible to add support for embedding tying with FSDP2 in Megatron?
Contributor guide
Assessment
This issue has not been assessed yet.