Intel Advanced Matrix Extensions (AMX) support
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 4.7k
- Forks
- 536
- Avg merge
- 12h 12m
- Merged PRs (30d)
- 4
Description
Hello CTranslate2 developers,
I am a user of your library and I appreciate your work on providing fast and accurate inference engine. I am wondering if you have any plans to support Intel Advanced Matrix Extensions (AMX) for CPU inference. According to Intel, AMX can speed up inference by several factors for certain models and data types.
I have tried to compile CTranslate2 from the source code with the -mamx-tile -mamx-int8 -mamx-bf16 flags, but it seems that there are some additional steps required to enable AMX (maybe adding a new kernel as vec_amx.h with modifiying vec_avx512.h to enable AMX tile operations).
I would appreciate it if you could share your thoughts on this topic and let me know if AMX support is feasible and desirable for CTranslate2.
Thank you for your time and attention.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the AMX compiler flags mentioned in the issue and the existing vec_avx512.h implementation. The proposed vec_amx.h file is another entry point to investigate, along with the additional steps needed for AMX tile operations. Done would mean defining and validating a feasible AMX path for CPU inference, but the issue does not specify target models, data types, tests, or acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- machine-learning, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100