tensorflow / tensorflow/models
How to reduce CPU usage?
Open
@laxmareddyp is already working on this.
Since Jul 27, 2023.
type:support
- Dominant language
- Python
- Stars
- 77.7k
- Forks
- 44.8k
- PR merge metrics
- No merged PRs in 30d
Description
I use this code to load the model:
model = 'WizardLM/WizardCoder-15B-V1.0'
def load_model(model = model):
tokenizer = AutoTokenizer.from_pretrained(model)
model = AutoModelForCausalLM.from_pretrained(model, device_map=device_map, load_in_8bit = True)
return tokenizer, model
tokenizer, model = load_model(model)
And this code to generate:
generation_config = GenerationConfig(
temperature=0.0,
top_p=0.95,
top_k=50,
eos_token_id=tokenizer.eos_token_id,
pad_token_id=tokenizer.pad_token_id,
)
prompt_template = f'''
Below is an instruction that describes a task. Write a response that appropriately completes the request
### Instruction: {prompt}
### Response:'''
inputs = tokenizer(prompt_template, return_tensors="pt").to("cuda")
generated_ids = model.generate(**inputs, generation_config=generation_config, max_new_tokens=6000)
outputs = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)
This model fit at all into my GPU, but for some reason the GPU not even used (it is not heating while generating), but proccesor usage is 100%
What`s wrong with my code or problem is in the model?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.