intel NPU is so slow
Open
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 40.3k
- Forks
- 3.7k
- PR merge metrics
- No merged PRs in 30d
Description
I used Intel's AI Boost, and it's too slow...,NPU uasge 37%
cd .\transformers-bitnet
pip install .
pip install torch accelerate intel-npu-acceleration-library gradio
import torch
import torch._dynamo
torch._dynamo.config.suppress_errors = True
from transformers import AutoModelForCausalLM, AutoTokenizer, TextStreamer
from intel_npu_acceleration_library import NPUModelForCausalLM
from intel_npu_acceleration_library.compiler import CompilerConfig
import gradio as gr
model_id = "microsoft/bitnet-b1.58-2B-4T"
tokenizer = AutoTokenizer.from_pretrained(model_id)
compiler_conf = CompilerConfig( dtype=torch.bfloat16)
# optimized_model = torch.compile(model_id, backend="npu")
model = NPUModelForCausalLM.from_pretrained(model_id, config=compiler_conf,use_cache=True,attn_implementation="sdpa").eval()
prompt = f"<s>用户:{user_input}\n助手:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
streamer = TextStreamer(tokenizer, skip_special_tokens=True)
outputs = model.generate(
**inputs,
do_sample=True,
temperature=0.5,
top_p=0.85,
repetition_penalty=1.1,
streamer=streamer,
top_k=50,
max_new_tokens=512
)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the listed installation commands and the Python entry point using NPUModelForCausalLM with microsoft/bitnet-b1.58-2B-4T. Measure generation speed and NPU utilization, then inspect the Intel NPU acceleration path and record a specific bottleneck or reproducible performance result as the outcome.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100