int8 lora always load from disk
Open
Potential Bug
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 7h
- Merged PRs (30d)
- 158
Description
lora loading is much slower than normal safetensor. almost 1/10 speed, maybe because of weight merging in a for loop. some lora like bernini's is a few G , too slow. can it be cached after first running?
Contributor guide
Research direction
No file, test, or entry point is named. Start by reproducing the slow int8 LoRA load, compare it with safetensor loading, and trace the weight-merging path; done means subsequent runs reuse cached data instead of loading the LoRA from disk and the reported slowdown is reduced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 45/100