Performance on arch linux compared to llama-cpp vulkan
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1.9k
- Forks
- 152
- Avg merge
- 4h 14m
- Merged PRs (30d)
- 11
Description
In its current state, GPT OSS 20B Q4_K_M generates less heat when running on llama-cpp vulkan (LM Studio) compared to lemonade (FLM Q4NX). For the same query, LM Studio (32768 context) keeps temperatures close to 50 degrees while FLM (8192 context( heats the laptop upto 65 degrees. This is on a ryzen ai 7 350, 32 GB laptop running arch linux. Also, the FLM model seems to hallucinate more than the Q4_K_M one. I am using AnythingLLM as a frontend.
The main USP of an NPU for most people would be an "always on" LLM that sits in the background and is used throughout the OS for things like shell assistance and document summaries, while keeping the CPU and GPU available for other tasks. I really hope fastflowlm becomes that library which can facilitate this use case. I'd be happy to run tests and share output.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No source files, tests, or entry points are named. Start by reproducing the same GPT OSS 20B Q4_K_M query on the Ryzen AI 7 350 under Arch Linux with LM Studio and FastFlowLM, recording temperatures, context size, generation output, and configuration. Done requires a reproducible comparison with enough output to identify the performance or quality difference.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- arch-linux, cpp
- Domain
- ai, operating-systems, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100