cos CPU ops deterministic not ensured
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
### 🐛 Describe the bug
```
import torch
import hashlib
import numpy as np
import time
import os
import sys
import ctypes
from ctypes import wintypes
def calculate_md5(tensor):
"""计算张量的MD5校验和,用于验证确定性"""
tensor_bytes = tensor.numpy().tobytes()
md5_hash = hashlib.md5(tensor_bytes).hexdigest()
return md5_hash
def reproduce_cos_determinism_issue():
# 固定随机种子,确保初始张量生成的确定性
torch.manual_seed(0)
np.random.seed(0)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.use_deterministic_algorithms(True)
# ========== 关键:调用cos前打印加载的动态库 ==========
# 生成CPU float32的测试张量
freqs = torch.randn((1024, 512), dtype=torch.float32, device="cpu")
# 记录第一次计算的MD5作为基准
# base_cos = freqs.sin()
base_cos = freqs.cos()
base_md5 = calculate_md5(base_cos)
print(f"\n基准MD5值: {base_md5}")
print("开始循环验证cos()算子的确定性...\n")
# 多次循环执行cos(),检测MD5是否一致
max_iterations = 1000
inconsistency_count = 0
for i in range(max_iterations):
current_cos = freqs.cos()
current_md5 = calculate_md5(current_cos)
if current_md5 != base_md5:
inconsistency_count += 1
print(f"【异常】第{i+1}次计算MD5不一致!")
print(f" 基准MD5: {base_md5}")
print(f" 当前MD5: {current_md5}")
diff = torch.abs(base_cos - current_cos)
max_diff_idx = torch.argmax(diff)
print(f" 最大差异值: {diff.max().item()}, 位置: {max_diff_idx}")
# break
if (i + 1) % 1000 == 0:
print(f"已执行{i+1}次,MD5均一致,当前时间: {time.ctime()}")
if inconsistency_count == 0:
print(f"\n完成{max_iterations}次循环,未检测到MD5不一致的情况")
if __name__ == "__main__":
# 打印环境信息
print("=== 环境信息 ===")
print(f"Python版本: {sys.version}")
print(f"PyTorch版本: {torch.__version__}")
print(f"CUDA是否可用: {torch.cuda.is_available()}")
print(f"系统平台: {sys.platform}")
print(f"CPU设备: {torch.device('cpu')}")
print(f"float32精度设置: {torch.get_default_dtype()}")
# 执行复现逻辑
reproduce_cos_determinism_issue()
```
which prints:
```
=== 环境信息 ===
Python版本: 3.9.19 (main, Jul 3 2024, 11:02:17)
[GCC 8.3.0]
PyTorch版本: 2.7.1+cpu
CUDA是否可用: False
系统平台: linux
CPU设备: cpu
float32精度设置: torch.float32
基准MD5值: 60083a9f5df3dadd4600d1bb0a96577d
开始循环验证cos()算子的确定性...
【异常】第1次计算MD5不一致!
基准MD5: 60083a9f5df3dadd4600d1bb0a96577d
当前MD5: d572ba7c66a7c5be7f1d0fbeacf1527d
最大差异值: 0.00015109777450561523, 位置: 49327
【异常】第2次计算MD5不一致!
基准MD5: 60083a9f5df3dadd4600d1bb0a96577d
当前MD5: d572ba7c66a7c5be7f1d0fbeacf1527d
最大差异值: 0.00015109777450561523, 位置: 49327
【异常】第3次计算MD5不一致!
基准MD5: 60083a9f5df3dadd4600d1bb0a96577d
当前MD5: d572ba7c66a7c5be7f1d0fbeacf1527d
最大差异值: 0.00015109777450561523, 位置: 49327
【异常】第4次计算MD5不一致!
基准MD5: 60083a9f5df3dadd4600d1bb0a96577d
当前MD5: d572ba7c66a7c5be7f1d0fbeacf1527d
最大差异值: 0.00015109777450561523, 位置: 49327
...
```
This case occasionally produce: in the first time `cos` CPU result (base_cos), MD5 different from the following times(current_cos), where the following md5 are stable and correct as the md5 in non-reproduced times, and the different md5 (base_cos) can vary in several executions. The probability is quite low (maybe 20%).
Please help fix this issue since `cos` CPU operator is widely used in rope calculation in LLM training...
### Versions
Collecting environment information...
PyTorch version: 2.7.1+cpu
Is debug build: False
CUDA used to build PyTorch: None
ROCM used to build PyTorch: N/A
OS: Debian GNU/Linux 10 (buster) (x86_64)
GCC version: (GCC) 9.3.0
Clang version: Could not collect
CMake version: version 3.18.0
Libc version: glibc-2.28
Python version: 3.9.19 (main, Jul 3 2024, 11:02:17) [GCC 8.3.0] (64-bit runtime)
Python platform: Linux-5.10.135.bsk.6-amd64-x86_64-with-glibc2.28
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
Is XPU available: False
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True
Caching allocator config: N/A
CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 57 bits virtual
CPU(s): 180
On-line CPU(s) list: 0-179
Thread(s) per core: 2
Core(s) per socket: 45
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 143
Model name: Intel(R) Xeon(R) Platinum 8457C
Stepping: 8
CPU MHz: 2600.000
BogoMIPS: 5200.00
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 48K
L1i cache: 32K
L2 cache: 2048K
L3 cache: 99840K
NUMA node0 CPU(s): 0-89
NUMA node1 CPU(s): 90-179
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid cldemote movdiri movdir64b fsrm md_clear serialize tsxldtrk arch_lbr amx_bf16 avx512_fp16 amx_tile amx_int8 arch_capabilities
Versions of relevant libraries:
[pip3] numpy==1.24.4
[pip3] torch==2.7.1+cpu
[pip3] torchvision==0.16.0+cpu
[pip3] triton==3.0.0
[conda] Could not collect
cc @ezyang @gchanan @kadeng @msaroufim @jgong5 @mingfeima @XiaobingSuper @sanchitintel @ashokei @jingxu10 @jerryzh168 @aditew01 @mruberry @kurtamohler
Contributor guide
Assessment
This issue has not been assessed yet.