Lightning-AI / Lightning-AI/litgpt
Cut cross entropy for pretraining and finetuning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 13.7k
- Forks
- 1.5k
- Avg merge
- 15h 37m
- Merged PRs (30d)
- 1
Description
[Cut cross entropy](https://arxiv.org/pdf/2411.09009) has been shown in my own work to be really easy to use and saves a lot of memory when training models with large vocab sizes. Therefore I think adding it to litGPT would help people fit bigger models more effortlessly. The main idea is that with some clever intermediate states we can avoid materializing the full logit tensor (Batch x sequence x vocab size), thus saving a lot of memory. The loss is instead computed using the final hidden states and the weight matrix.
As to how this can be integrated, I think here's how it can happen:
1. The GPT model needs an option to return not the logits but the output projection weight and the final hiddens.
2. An option can be added to use for the cross entropy for pretrain and finetune script.
I'm happy to help with implementing this if folks think this is a worthy addition to litGPT.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the linked cut-cross-entropy paper, then inspect the GPT model and the pretraining and finetuning scripts. Trace how final hidden states, the output projection weight, and logits currently flow. Done means both scripts can opt into the proposed loss while avoiding materialization of the full logits tensor.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100