pytorch / pytorch/executorch

`torchao:` quantization schemes can't find ao lib

Open
#8,937 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

module: quantization triaged
Dominant language
Python
Stars
5k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
581

Description

🐛 Describe the bug

Anytime _load_torchao_aten_lib is called (namely for -qmode 8da[]w), the following error pops up:

Traceback (most recent call last):
File "/home/jackzhxng/.conda/envs/oss2/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/jackzhxng/.conda/envs/oss2/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/home/jackzhxng/torchrepos2/executorch/examples/models/llama/runner/eager.py", line 116, in
main() # pragma: no cover
File "/home/jackzhxng/torchrepos2/executorch/examples/models/llama/runner/eager.py", line 112, in main
execute_runner(EagerLlamaRunner)
File "/home/jackzhxng/torchrepos2/executorch/examples/models/llama/runner/eager.py", line 93, in execute_runner
runner = runner_class(args) # pyre-ignore: Missing argument [20]
File "/home/jackzhxng/torchrepos2/executorch/examples/models/llama/runner/eager.py", line 38, in init
manager: LLMEdgeManager = _prepare_for_llama_export(args)
File "/home/jackzhxng/.conda/envs/oss2/lib/python3.10/site-packages/executorch/examples/models/llama/export_llama_lib.py", line 601, in _prepare_for_llama_export
.source_transform(_get_source_transforms(args.model, dtype_override, args))
File "/home/jackzhxng/.conda/envs/oss2/lib/python3.10/site-packages/executorch/extension/llm/export/builder.py", line 154, in source_transform
self.model = transform(self.model)
File "/home/jackzhxng/.conda/envs/oss2/lib/python3.10/site-packages/executorch/examples/models/llama/source_transformation/quantize.py", line 101, in quantize
_load_torchao_aten_lib(libname="libtorchao_ops_aten")
File "/home/jackzhxng/.conda/envs/oss2/lib/python3.10/site-packages/executorch/examples/models/llama/source_transformation/quantize.py", line 823, in _load_torchao_aten_lib
assert (
AssertionError: Expected 1 library but got 0. If you installed the torchao ops in a non-standard location, please set CMAKE_INSTALL_PREFIX correctly.

Versions

PyTorch version: 2.7.0.dev20250131+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A

OS: CentOS Stream 9 (x86_64)
GCC version: (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2)
Clang version: Could not collect
CMake version: version 3.30.5
Libc version: glibc-2.34

Python version: 3.10.0 (default, Mar 3 2022, 09:58:08) [GCC 7.5.0] (64-bit runtime)
Python platform: Linux-6.4.3-0_fbk12_hardened_2624_g7d95a0297d81-x86_64-with-glibc2.34
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 40 bits physical, 57 bits virtual
Byte Order: Little Endian
CPU(s): 166
On-line CPU(s) list: 0-165
Vendor ID: AuthenticAMD
Model name: AMD EPYC-Milan Processor
CPU family: 25
Model: 1
Thread(s) per core: 2
Core(s) per socket: 83
Socket(s): 1
Stepping: 1
BogoMIPS: 2399.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy svm cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single ssbd ibrs ibpb stibp vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx512_bf16 clzero xsaveerptr wbnoinvd arat npt lbrv nrip_save tsc_scale vmcb_clean pausefilter pfthreshold avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm arch_capabilities
Virtualization: AMD-V
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 2.6 MiB (83 instances)
L1i cache: 2.6 MiB (83 instances)
L2 cache: 41.5 MiB (83 instances)
L3 cache: 32 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-165
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec store bypass: Vulnerable
Vulnerability Spectre v1: Vulnerable: __user pointer sanitization and usercopy barriers only; no swapgs barriers
Vulnerability Spectre v2: Vulnerable, IBPB: disabled, STIBP: disabled, PBRSB-eIBRS: Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

Versions of relevant libraries:
[pip3] executorch==0.6.0a0+bb60a20
[pip3] flake8==6.1.0
[pip3] flake8-breakpoint==1.1.0
[pip3] flake8-bugbear==24.4.26
[pip3] flake8-comprehensions==3.14.0
[pip3] flake8-plugin-utils==1.3.3
[pip3] flake8-pyi==23.5.0
[pip3] mypy==1.14.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==2.2.2
[pip3] torch==2.7.0.dev20250131+cpu
[pip3] torchao==0.10.0+git7d879462
[pip3] torchaudio==2.6.0.dev20250131+cpu
[pip3] torchsr==1.0.4
[pip3] torchtune==0.0.0
[pip3] torchvision==0.22.0.dev20250131+cpu
[conda] executorch 0.6.0a0+bb60a20 pypi_0 pypi
[conda] numpy 2.2.2 pypi_0 pypi
[conda] torch 2.7.0.dev20250131+cpu pypi_0 pypi
[conda] torchao 0.10.0+git7d879462 pypi_0 pypi
[conda] torchaudio 2.6.0.dev20250131+cpu pypi_0 pypi
[conda] torchfix 0.6.0 pypi_0 pypi
[conda] torchsr 1.0.4 pypi_0 pypi
[conda] torchtune 0.0.0 pypi_0 pypi
[conda] torchvision 0.22.0.dev20250131+cpu pypi_0 pypi

cc @kimishpatel @jerryzh168

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in examples/models/llama/source_transformation/quantize.py at _load_torchao_aten_lib, especially the call around line 811 and the assertion around line 823. Reproduce the -qmode 8da[]w path and inspect how the torchao library location and CMAKE_INSTALL_PREFIX are discovered. Done means the quantization scheme finds the expected library instead of reporting zero libraries.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, python
Domain
machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.