intra-doc causal masking
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 3h
- Merged PRs (30d)
- 272
Description
**Is your feature request related to a problem? Please describe.**
Currently it is not possible to pretrained a model with document masking, while it has demonstrated better performance, better long context capability and potentially massive flops reduction.
Intra-doc masking will also greatly benefit Post-Training as current SFT consist of padding up to the sequence length, with Doc masking, it will greatly improve the throughput of SFT as conversation can now be merge inside one sequence.
**Describe the solution you'd like**
Add an argument :
```
--intra-doc-masking
```
that would apply an intra doc masking
**Additional context**
It would require update into Transformer Engine, to use `flash_attn_with_kvcache` or develloping something specific in cuDNN
Contributor guide
Research direction
The issue names no files or tests; start by locating argument parsing and the attention-mask path for pretraining and SFT. Investigate the required Transformer Engine integration, including flash_attn_with_kvcache or a cuDNN implementation, and define completion as --intra-doc-masking applying the intended mask in both workflows with validation of throughput and correctness.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100