flagos-ai / flagos-ai/FlagBLAS
dotc benchmark ci fail
- Dominant language
- Python
- Stars
- 3
- Forks
- 7
- Avg merge
- 11h 54m
- Merged PRs (30d)
- 38
Description
platform: h100
error info:
=========================== short test summary info ============================
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 610146 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_cdotc_stride[2-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 610146 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_cdotc_stride[2-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 6.44 GiB is free. Process 610146 has 72.66 GiB memory in use. Of the allocated memory 72.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_cdotc_stride[3-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 610146 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_cdotc_stride[3-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 6.44 GiB is free. Process 610146 has 72.66 GiB memory in use. Of the allocated memory 72.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc_stride[2-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 30.44 GiB is free. Process 610146 has 48.66 GiB memory in use. Of the allocated memory 48.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc_stride[2-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 46.44 GiB is free. Process 610146 has 32.66 GiB memory in use. Of the allocated memory 32.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc_stride[3-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 46.44 GiB is free. Process 610146 has 32.66 GiB memory in use. Of the allocated memory 32.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc_stride[3-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 30.44 GiB is free. Process 610146 has 48.66 GiB memory in use. Of the allocated memory 48.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
================== 9 failed, 1 passed, 140 warnings in 10.54s ==================
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with benchmark/test_dotc_perf.py and run the failing dotc performance tests on the H100 setup described in the issue. Investigate the reported CUDA out-of-memory failures and determine whether the benchmark or CI configuration needs adjustment; done means the affected tests complete without out-of-memory errors.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- ci-cd, performance, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100