pytorch / pytorch/pytorch

INTERNAL ASSERT FAILED at "/pytorch/c10/core/AllocatorConfig.h":124 on python torch import

Open
#186,453 7 comments 0 reactions 0 assignees View on GitHub
actionable module: crash module: CUDACachingAllocator triaged
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🐛 Describe the bug

I had torch successfully run for a while, and then after one program crash I cannot run it afterwards, constantly giving me this error:
```
terminate called after throwing an instance of 'c10::Error'
what(): i < config_.size() INTERNAL ASSERT FAILED at "/pytorch/c10/core/AllocatorConfig.h":124, please report a bug to PyTorch. Index out of bounds in ConfigTokenizer
Exception raised from checkIndex at /pytorch/c10/core/AllocatorConfig.h:124 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string, std::allocator >) + 0x9d (0x7f29e116afad in /home/fcwe1113/PycharmProjects/ASCEND/.venv/lib/python3.13/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string, std::allocator > const&) + 0xca (0x7f29e10f9481 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/lib/python3.13/site-packages/torch/lib/libc10.so)
frame #2: c10::detail::torchInternalAssertFail(char const*, char const*, unsigned int, char const*, char const*) + 0x42 (0x7f29e11652b2 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/lib/python3.13/site-packages/torch/lib/libc10.so)
frame #3: + 0x2ac2e (0x7f29e1590c2e in /home/fcwe1113/PycharmProjects/ASCEND/.venv/lib/python3.13/site-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x61ef (0x7f2a256141ef in /lib64/ld-linux-x86-64.so.2)
frame #5: + 0x62cd (0x7f2a256142cd in /lib64/ld-linux-x86-64.so.2)
frame #6: _dl_catch_exception + 0x157 (0x7f2a25610517 in /lib64/ld-linux-x86-64.so.2)
frame #7: + 0xd359 (0x7f2a2561b359 in /lib64/ld-linux-x86-64.so.2)
frame #8: _dl_catch_exception + 0xa9 (0x7f2a25610469 in /lib64/ld-linux-x86-64.so.2)
frame #9: + 0xd82a (0x7f2a2561b82a in /lib64/ld-linux-x86-64.so.2)
frame #10: + 0x94634 (0x7f2a25294634 in /usr/lib/libc.so.6)
frame #11: _dl_catch_exception + 0xa9 (0x7f2a25610469 in /lib64/ld-linux-x86-64.so.2)
frame #12: + 0x25b9 (0x7f2a256105b9 in /lib64/ld-linux-x86-64.so.2)
frame #13: + 0x94123 (0x7f2a25294123 in /usr/lib/libc.so.6)
frame #14: dlopen + 0x6b (0x7f2a252946eb in /usr/lib/libc.so.6)
frame #15: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x178790a]
frame #16: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x1787867]
frame #17: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x1787814]
frame #18: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x1604536]
frame #19: _PyEval_EvalFrameDefault + 0xb8a5 (0x162356b in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #20: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x16af0c2]
frame #21: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x16aee8e]
frame #22: PyObject_CallMethodObjArgs + 0xa8 (0x16aece8 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #23: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x18c10b9]
frame #24: _PyEval_EvalFrameDefault + 0xdea2 (0x1625b68 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #25: PyEval_EvalCode + 0x94 (0x16a8454 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #26: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x16a8d48]
frame #27: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x1605367]
frame #28: _PyEval_EvalFrameDefault + 0xe812 (0x16264d8 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #29: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x16af0c2]
frame #30: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x16aee8e]
frame #31: PyObject_CallMethodObjArgs + 0xa8 (0x16aece8 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #32: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x18c10b9]
frame #33: _PyEval_EvalFrameDefault + 0xdea2 (0x1625b68 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #34: PyEval_EvalCode + 0x94 (0x16a8454 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #35: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x16a8d48]
frame #36: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x1605367]
frame #37: _PyEval_EvalFrameDefault + 0x9ce6 (0x16219ac in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #38: PyEval_EvalCode + 0x94 (0x16a8454 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #39: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x16a186a]
frame #40: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x16ca443]
frame #41: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x17d5d47]
frame #42: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x17d5577]
frame #43: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x173b010]
frame #44: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x173afb6]
frame #45: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x173aeaa]
frame #46: Py_RunMain + 0x340 (0x1781a92 in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
frame #47: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x1781745]
frame #48: /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python() [0x178152d]
frame #49: + 0x27741 (0x7f2a25227741 in /usr/lib/libc.so.6)
frame #50: __libc_start_main + 0x89 (0x7f2a25227879 in /usr/lib/libc.so.6)
frame #51: _start + 0x29 (0x18092fb in /home/fcwe1113/PycharmProjects/ASCEND/.venv/bin/python)
```
note that this happens on torch import, I was confused for a second why I still see the above error when the breakpoint i set is on the first line in __main__, setting the breakpoint right before ```import torch``` confirmed this

### Versions

```
Collecting environment information...
PyTorch version: N/A
Is debug build: N/A
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: N/A

OS: Arch Linux (x86_64)
GCC version: (GCC) 16.1.1 20260430
Clang version: 22.1.6
CMake version: version 4.3.3
Libc version: glibc-2.43

Python version: 3.14.5 (main, May 10 2026, 18:26:20) [GCC 16.1.1 20260430] (64-bit runtime)
Python platform: Linux-7.0.11-arch1-1-x86_64-with-glibc2.43
Is CUDA available: N/A
CUDA runtime version: 13.3.33
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: GPU 0: NVIDIA GeForce RTX 4050 Laptop GPU
Nvidia driver version: Could not collect
cuDNN version: Could not collect
Is XPU available: N/A
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: N/A
Caching allocator config: N/A

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 48 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: AuthenticAMD
Model name: AMD Ryzen 7 7435HS
CPU family: 25
Model: 68
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 1
Microcode version: 0xa404108
Frequency boost: enabled
CPU(s) scaling MHz: 53%
CPU max MHz: 4553.9829
CPU min MHz: 411.5050
BogoMIPS: 6188.14
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local user_shstk clzero irperf xsaveerptr rdpru wbnoinvd cppc arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm debug_swap
Virtualization: AMD-V
L1d cache: 256 KiB (8 instances)
L1i cache: 256 KiB (8 instances)
L2 cache: 4 MiB (8 instances)
L3 cache: 16 MiB (1 instance)
NUMA node(s): 1
NUMA node0 CPU(s): 0-15
Vulnerability Gather data sampling: Not affected
Vulnerability Ghostwrite: Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Mmio stale data: Not affected
Vulnerability Old microcode: Not affected
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed: Not affected
Vulnerability Spec rstack overflow: Mitigation; Safe RET
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Retpolines; IBPB conditional; IBRS_FW; STIBP always-on; RSB filling; PBRSB-eIBRS Not affected; BHI Not affected
Vulnerability Srbds: Not affected
Vulnerability Tsa: Mitigation; Clear CPU buffers
Vulnerability Tsx async abort: Not affected
Vulnerability Vmscape: Mitigation; IBPB before exit to userspace

Versions of relevant libraries:
[pip3] numpy==2.4.6
[conda] Could not collect

```

```uv``` reported the torch version to be ```2.12.0```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.