flagos-ai / flagos-ai/FlagBLAS
dotu benchmark ci fail
- Dominant language
- Python
- Stars
- 3
- Forks
- 7
- Avg merge
- 11h 54m
- Merged PRs (30d)
- 38
Description
platform: h100
error info:
=========================== short test summary info ============================
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 2000575 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_cdotu_stride[2-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 2000575 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_cdotu_stride[2-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 6.44 GiB is free. Process 2000575 has 72.66 GiB memory in use. Of the allocated memory 72.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_cdotu_stride[3-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 2000575 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_cdotu_stride[3-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 6.44 GiB is free. Process 2000575 has 72.66 GiB memory in use. Of the allocated memory 72.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu_stride[2-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 30.44 GiB is free. Process 2000575 has 48.66 GiB memory in use. Of the allocated memory 48.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu_stride[2-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 46.44 GiB is free. Process 2000575 has 32.66 GiB memory in use. Of the allocated memory 32.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu_stride[3-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 46.44 GiB is free. Process 2000575 has 32.66 GiB memory in use. Of the allocated memory 32.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu_stride[3-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 30.44 GiB is free. Process 2000575 has 48.66 GiB memory in use. Of the allocated memory 48.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with benchmark/test_dotu_perf.py and the H100 CI setup, then run the reported dotu and cdotu performance tests to reproduce the PyTorch CUDA out-of-memory failures. Determine why these cases exceed available GPU memory; done means the benchmark CI completes without these failures.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ci-cd, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100