deepjavalibrary / deepjavalibrary/djl
Multi lora adapter inference advanced document seems wrong
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 760
- Avg merge
- 19h 26m
- Merged PRs (30d)
- 17
Description
## Description
Multi lora adapter inference advanced seems wrong: https://docs.djl.ai/master/docs/demos/aws/sagemaker/large-model-inference/sample-llm/multi_lora_adapter_inference_advanced.html
In the evaluate function, it passes adapters as an input to model.generate(), which does not contain any such argument, going through the transformers github.
Also as mentioned in the PEFT load_adapter function, we need to set_adapter in order to set the active adapter, which is not done in the document.
After running the example on my code, I was getting incorrect results. On debugging later, I found this issue.
Can you help me with the correct code to run multi-lora adapter inference.
Contributor guide
Research direction
Start with the advanced multi-LoRA inference document at the linked URL and inspect its evaluate function. Compare the model.generate call with the transformers API and the PEFT load_adapter and set_adapter guidance named in the report. Done means the document contains correct code for selecting the active adapter and running multi-LoRA inference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, machine-learning
- Issue type
- Documentation
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100