deepjavalibrary / deepjavalibrary/djl

Multi lora adapter inference advanced document seems wrong

Open
#3,583 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
4.9k
Forks
760
Avg merge
19h 26m
Merged PRs (30d)
17

Description

## Description
Multi lora adapter inference advanced seems wrong: https://docs.djl.ai/master/docs/demos/aws/sagemaker/large-model-inference/sample-llm/multi_lora_adapter_inference_advanced.html

In the evaluate function, it passes adapters as an input to model.generate(), which does not contain any such argument, going through the transformers github.

Also as mentioned in the PEFT load_adapter function, we need to set_adapter in order to set the active adapter, which is not done in the document.

After running the example on my code, I was getting incorrect results. On debugging later, I found this issue.

Can you help me with the correct code to run multi-lora adapter inference.

Contributor guide

Open the contributing guide

Research direction

Start with the advanced multi-LoRA inference document at the linked URL and inspect its evaluate function. Compare the model.generate call with the transformers API and the PEFT load_adapter and set_adapter guidance named in the report. Done means the document contains correct code for selecting the active adapter and running multi-LoRA inference.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, machine-learning
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.