Flash attention for fine-tuning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 39.5k
- Forks
- 4.8k
- PR merge metrics
- No merged PRs in 30d
Description
How can we use flash attention v2 for fine-tuning with huggingface models?
Does the path only works for pre-training(or extended pre-training)?
- [Link](https://github.com/pacman100/DHS-LLM-Workshop/blob/53672e1b774da7798fb10a50ef8ca5b2750c5608/personal_copilot/training/starcoder_flash_attn_monkey_patch.py#L11)
All the discussions mentioned below are for pre-training(or extended pre-training).
- https://github.com/huggingface/accelerate/issues/1864
- https://github.com/pacman100/DHS-LLM-Workshop/issues/1
I would like to fine-tune `bigcode/starcoder` 15.5 billion parameter model with 2k context length using A100-80GB.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked starcoder_flash_attn_monkey_patch.py reference and the related Accelerate and DHS-LLM-Workshop discussions. Determine whether FastChat has a fine-tuning entry point for bigcode/starcoder and what changes are required for flash attention v2 with a 2k context on an A100-80GB; the issue is done only when that supported path and its validation are defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100