deepspeedai / deepspeedai/DeepSpeedExamples
About multiple-thread attention computation on CPU using zero-inference example.
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.8k
- Forks
- 1.1k
- Avg merge
- 2d 16h
- Merged PRs (30d)
- 1
Description
Hi,
I am trying to test the attention computation on the CPU with zero-interference.
I use the following command to run the script.
BSZ=96
LOG_DIR=$BASE_LOG_DIR/${MODEL_NAME}_bs${BSZ}
mkdir -p $LOG_DIR
deepspeed --num_gpus 1 run_model.py --dummy --model ${FULL_MODEL_NAME} --batch-size ${BSZ} --cpu-offload --pin-memory 1 --offload-dir /tmp/data/dxu --gen-len 32 --pin-memory 1 --kv-offload --async_kv_offload
Duing the exection, I only saw one core is used.
However, there are many processes are created for the run.
Are there any other parameters or env I should configure to enable multiple-cores?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with run_model.py and the supplied DeepSpeed command, focusing on the CPU offload, pin-memory, KV offload, and async KV offload options. Reproduce the zero-interference run and inspect process and core usage; done means identifying the missing configuration or confirming that the example cannot use multiple CPU cores.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100