facebookresearch/xformers

View on GitHub

Hackable and optimized Transformers building blocks, supporting a composable construction.

Stars: 10.5k
Forks: 787
Open beginner issues: 0
Indexed issues: 364
Avg merge: 9d 10h
Merged PRs (30d): 1

Dominant language: Python
License: No license data
Last GitHub push: Sep 10, 2026
Latest indexed: Sep 15, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: good first issue

Open All

364 open issues indexed

[perf] Auto causal <> sparse Open

enhancement

facebookresearch/xformers#18 · 0 comments · 0 reactions · 2 assignees · Indexed Sep 15, 2026
[block factory] Handle attentions which return one or several extra context tensors Open

facebookresearch/xformers#23 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
[LRA] Use key masking instead of attention mask Open

facebookresearch/xformers#38 · 0 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Support shape types for Tensors. Open

facebookresearch/xformers#43 · 0 comments · 1 reaction · 1 assignee · Indexed Sep 15, 2026
Support Tensor parallel or pipeline parallel out of the box Open

brainstorm

facebookresearch/xformers#70 · 8 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
[feat] patch torch and huggingface for improved performance in existing applications Open

facebookresearch/xformers#78 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
[improvement] lower Favor+causal memory consumption Open

blocked enhancement

facebookresearch/xformers#105 · 3 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
[follow up] Check Nystrom + causal Open

facebookresearch/xformers#107 · 2 comments · 3 reactions · 1 assignee · Indexed Sep 15, 2026
Add conda package Open

blocked

facebookresearch/xformers#152 · 30 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
Add LRA to CI Open

facebookresearch/xformers#155 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
pip install broken on OSX and Ubuntu 20.04 (container w/ CUDA) python 3.9 Open

facebookresearch/xformers#157 · 22 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
xformers ViT-B ImageNet MAE + Deepnorm training instability Open

ongoing

facebookresearch/xformers#219 · 62 comments · 1 reaction · 1 assignee · Indexed Sep 15, 2026
JIT scripting is broken Open

facebookresearch/xformers#246 · 11 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Assertion statements in attention implementation Open

facebookresearch/xformers#264 · 9 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Rotary Embeddings + Triton Open

ongoing

facebookresearch/xformers#265 · 12 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
Support for attention masks in blocksparse layers Open

facebookresearch/xformers#289 · 8 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Hydra ConfigStore doesn't store the registered name in the stored node Open

facebookresearch/xformers#332 · 2 comments · 0 reactions · 1 assignee · Indexed Sep 15, 2026
Flexible GATE in MOE layer Open

facebookresearch/xformers#350 · 2 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
[RFC] Refactor and simplify `scaled_dot_product_attention` for easier extensibility Open

brainstorm

facebookresearch/xformers#351 · 3 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
Addition of LaTr: Layout-aware transformer for scene-text VQA Open

facebookresearch/xformers#361 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
fatal error C1083: Cannot open compiler generated file: '': Invalid argument Open

facebookresearch/xformers#365 · 8 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
Replace `vec_t*` by `const vec_t*` in attention.cu Open

facebookresearch/xformers#367 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Torch JIT breaks when memory_efficient_attention Open

bug

facebookresearch/xformers#406 · 13 comments · 3 reactions · 0 assignees · Indexed Sep 15, 2026
Could not run 'xformers::efficient_attention_forward_generic' with arguments from the 'CUDA' backend - USE_MEMORY_EFFICIENT_ATTENTION over gunicorn Open

facebookresearch/xformers#457 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
recursion error reached hipifying cutlass portion of xformers Open

facebookresearch/xformers#485 · 3 comments · 3 reactions · 0 assignees · Indexed Sep 15, 2026
-fstack-clash-protection'; did you mean '-fstack-protector'? (and -fcf-protection) Open

facebookresearch/xformers#497 · 9 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Whether windwos is supported? Open

facebookresearch/xformers#501 · 12 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
[xformers/benchmarks/benchmark_encoder.py] fatal error: cuda.h: No such file or directory + AttributeError: module 'triton' has no attribute 'code_gen' Open

facebookresearch/xformers#516 · 6 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Support for `pip` Open

facebookresearch/xformers#532 · 1 comment · 4 reactions · 0 assignees · Indexed Sep 15, 2026
Support pure pytorch implementation for memory_efficient_attention Open

facebookresearch/xformers#536 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Windows build instructions fail because of git submodule checkout issues Open

facebookresearch/xformers#543 · 7 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
6 tests failing in test_core_attention, Open

facebookresearch/xformers#544 · 10 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
fail test_pytorch_transformer_parity.py::test_pytorch_encoder_parity - assert 85.07492282178951 > 120 Open

facebookresearch/xformers#546 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
JVP/forward mode AD support for flash attention? Open

facebookresearch/xformers#552 · 4 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Can't install xformers via poetry: ParseConstraintError Could not parse version constraint: github.com/facebookresearch/xformers.git@main#egg=xformers Open

facebookresearch/xformers#553 · 3 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
`No operator found for this attention` when `attn_bias` is a `torch.Tensor` Open

facebookresearch/xformers#576 · 12 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Unable to Build from latest Open

facebookresearch/xformers#604 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
[Windows] Build from source fails with `Filename too long error: unable to create file ` Open

facebookresearch/xformers#605 · 17 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
linux install fatal error: cusolverDn.h Open

facebookresearch/xformers#607 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
[Question / Docs] Clip Speed up Open

facebookresearch/xformers#615 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Allowing decoder only definition Open

facebookresearch/xformers#617 · 2 comments · 2 reactions · 0 assignees · Indexed Sep 15, 2026
How to set random seeds fixed Open

good first issue

facebookresearch/xformers#618 · 17 comments · 1 reaction · 0 assignees · Indexed Sep 15, 2026
Couldnt get it installed on kaggle Open

facebookresearch/xformers#622 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
LNK2001: unresolved external symbol when building on Windows Open

facebookresearch/xformers#625 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Questions on speeding up ViT Open

facebookresearch/xformers#626 · 14 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
[RTX 3090] Raise NotImplementedError: No operator found for this attention: Inputs when I backward the loss Open

bug

facebookresearch/xformers#628 · 13 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Provide an easy way to debug mem-eff issues Open

good first issue

facebookresearch/xformers#632 · 0 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
torch.cuda.is_available() returns false after installing xformers Open

facebookresearch/xformers#634 · 1 comment · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Proposal for separate passing of causal flag and other types of bias Open

facebookresearch/xformers#640 · 5 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026
Local attention is slower and uses more memory than flash attention Open

facebookresearch/xformers#644 · 3 comments · 0 reactions · 0 assignees · Indexed Sep 15, 2026

facebookresearch/xformers

Get new issues in your inbox