flagos-ai / flagos-ai/FlagBLAS

dotu benchmark ci fail

Open
#35 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3
Forks
7
Avg merge
11h 54m
Merged PRs (30d)
38

Description

platform: h100
error info:
=========================== short test summary info ============================
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 2000575 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_cdotu_stride[2-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 2000575 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_cdotu_stride[2-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 6.44 GiB is free. Process 2000575 has 72.66 GiB memory in use. Of the allocated memory 72.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_cdotu_stride[3-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 16.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 14.44 GiB is free. Process 2000575 has 64.66 GiB memory in use. Of the allocated memory 64.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_cdotu_stride[3-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 24.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 6.44 GiB is free. Process 2000575 has 72.66 GiB memory in use. Of the allocated memory 72.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu_stride[2-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 32.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 30.44 GiB is free. Process 2000575 has 48.66 GiB memory in use. Of the allocated memory 48.00 GiB is allocated by PyTorch, and 2.00 MiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu_stride[2-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 46.44 GiB is free. Process 2000575 has 32.66 GiB memory in use. Of the allocated memory 32.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu_stride[3-2] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 46.44 GiB is free. Process 2000575 has 32.66 GiB memory in use. Of the allocated memory 32.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
FAILED benchmark/test_dotu_perf.py::test_perf_zdotu_stride[3-3] - torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 48.00 GiB. GPU 0 has a total capacity of 79.11 GiB of which 30.44 GiB is free. Process 2000575 has 48.66 GiB memory in use. Of the allocated memory 48.00 GiB is allocated by PyTorch, and 0 bytes is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with benchmark/test_dotu_perf.py and the H100 CI setup, then run the reported dotu and cdotu performance tests to reproduce the PyTorch CUDA out-of-memory failures. Determine why these cases exceed available GPU memory; done means the benchmark CI completes without these failures.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
ci-cd, performance, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.