[PERF] MM models. Transfer images main->worker in low precision
Open
Nobody has claimed this yet.
Investigating
Performance
triaged
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Problem
After #6004 we still has sufficient overhead for image preparation.
Profile looks like
send is around 15% from image preparation time.
Solution
I noticed that we transfer image in fp32 format. Every pixel takes 3*sizeof(fp32)=12 bytes. If we transfer(and maybe can process) images in int8 it might sufficiently reduce send overhead. (or at least bf16 - anyway the model run in bf16 and we anyway convert the image to bf16).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.