vt CPU attention: the per-element dtype dispatch is 64.6% of the kernel, and 219 call sites share the defect
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 423
- Forks
- 53
- Avg merge
- 20h 26m
- Merged PRs (30d)
- 310
Description
Row: VT-CPU-ELEM-DISPATCH
vt::AttentionCross and vt::Attention spend most of their CPU time resolving
an element type and a byte offset, not doing arithmetic.
Both CPU kernels (src/vt/cpu/cpu_ops.cpp) read every operand element through
LoadF32, which switches on t.dtype and multiplies the offset by
vt::SizeOf(t.dtype) once per element. SizeOf was defined out of line in
src/vt/dtype.cpp and the build enables no LTO — CMakeLists.txt sets no
INTERPROCEDURAL_OPTIMIZATION and passes no -flto — so it was a cross-
translation-unit call per element. AttentionCrossKernel's score loop calls
LoadF32 twice per element, around a body that is one multiply and one add.
A perf profile of vt::AttentionCross alone, at the LTX-2.5 connector's
video shape (Tq=S=1024, H=32, D=128, f32), on this x86-64 AVX-512 devbox:
| symbol | self |
|---|---|
LoadF32(Tensor const&, long) |
36.14% |
vt::SizeOf(vt::DType) |
28.41% |
AttentionCrossKernel(...)::{lambda(long, long)#1} |
15.60% |
64.6% of the kernel's own CPU time resolves an element type and an address; the
arithmetic and the softmax are the 15.60% line. The kernel runs at ~3.6 GFLOP/s
on 20 threads.
This is not an LTX-2.5 problem. LoadF32/StoreF32 appear at 219 call sites
across 64 kernels in src/vt/cpu/cpu_ops.cpp and src/vt/cpu/cpu_paged_attn.cpp;
attention is only where it was measured. Every CPU model path pays it.
The repair is the transformation MatmulOneChunk already applies against
MatmulOneChunkRef in the same file: resolve the element type once, outside the
loops, and walk typed pointers. It touches no output's accumulation order, so it
is bit-exact rather than close.
Measured on dgx-class hardware by the predecessor row
(.agents/specs/ltx25-connector-gemm.md, ## Owed), and reproduced here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in src/vt/cpu/cpu_ops.cpp and src/vt/cpu/cpu_paged_attn.cpp, using MatmulOneChunk and MatmulOneChunkRef as the stated precedent. Audit the 219 LoadF32/StoreF32 call sites and the Attention CPU kernels, then verify that element types and offsets are no longer resolved per element while accumulation remains bit-exact.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- performance
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100