flagos-ai / flagos-ai/FlagBLAS

dotc benchmark ci fail

Open
#31 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
7
Avg merge
11h 54m
Merged PRs (30d)
38

Description

platform: h100
error info:
=========================== short test summary info ============================
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 610146 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_cdotc_stride[2-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 610146 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_cdotc_stride[2-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 6.44 GiB is free. Process 610146 has 72.66 GiB memory in use. Of the allocated memory 72.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_cdotc_stride[3-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 610146 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_cdotc_stride[3-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 6.44 GiB is free. Process 610146 has 72.66 GiB memory in use. Of the allocated memory 72.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc_stride[2-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 30.44 GiB is free. Process 610146 has 48.66 GiB memory in use. Of the allocated memory 48.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc_stride[2-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 46.44 GiB is free. Process 610146 has 32.66 GiB memory in use. Of the allocated memory 32.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc_stride[3-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 46.44 GiB is free. Process 610146 has 32.66 GiB memory in use. Of the allocated memory 32.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotc_perf.py::test_perf_zdotc_stride[3-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 30.44 GiB is free. Process 610146 has 48.66 GiB memory in use. Of the allocated memory 48.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
================== 9 failed, 1 passed, 140 warnings in 10.54s ==================

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with benchmark/test_dotc_perf.py and run the failing dotc performance tests on the H100 setup described in the issue. Investigate the reported CUDA out-of-memory failures and determine whether the benchmark or CI configuration needs adjustment; done means the affected tests complete without out-of-memory errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ci-cd, performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.