OptimalScale / OptimalScale/LMFlow
Causal LM finetuning
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 8.5k
- Forks
- 822
- PR merge metrics
- No merged PRs in 30d
Description
Hey,
Great to see LISA implemented here.
As for the background, I am trying to finetune models with LORA other techniques on domain data but the Task i am doing is Causal LM is Next word Prediction.
Earlier I have been using DataCollator from Hugging face to convert the data into required format and use Trainer to train the model using LORA.
I have been quite curious for LISA , and wanted to do the same with LISA.
I am not getting the right process to convert my text into CausalLM data type and then use that to Finetune with LISA technique.
Any assistance/examples or references would be of great help.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the repository's LISA implementation and the existing causal language-model finetuning entry points. The issue does not name files, tests, or a defined completion criterion; clarify the desired data-conversion flow and example before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100