intel / intel/torch-xpu-ops

[release/2.14] [Bug Skip]: torch-xpu-ops.test.xpu.test_cuda_multigpu_xpu.TestCudaMultiGPU.test_mem_get_info failed at assert error not less thank expected value

Open
#4,775 1 comment 0 reactions 0 assignees View on GitHub
os: Linux os: Windows skipped test: ut wontfix
Dominant language
Python
Stars
113
Forks
128
Avg merge
5d 9h
Merged PRs (30d)
112

Description

### 🐛 Describe the bug with skip template

AssertionError: 68702699520 not less than 68702699520 To execute this test,

## Affected Tests

Cases:
~~op_ut,third_party.torch-xpu-ops.test.xpu.test_cuda_multigpu_xpu.TestCudaMultiGPU,test_mem_get_info~~

## Error Message

```

Traceback (most recent call last): File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/test_cuda_multigpu_xpu.py", line 1122, in test_mem_get_info _test(0) File "/__w/torch-xpu-ops/torch-xpu-ops/pytorch/third_party/torch-xpu-ops/test/xpu/test_cuda_multigpu_xpu.py", line 1118, in _test self.assertLess(after_free_bytes, before_free_bytes) File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/unittest/case.py", line 1232, in assertLess self.fail(self._formatMessage(msg, standardMsg)) File "/tmp/xpu-tool/Python/3.10.20/x64/lib/python3.10/unittest/case.py", line 675, in fail raise self.failureException(msg) AssertionError: 68702699520 not less than 68702699520 To execute this test, run the following from the base repo dir: PYTORCH_TEST_WITH_SLOW=1 python test/xpu/test_cuda_multigpu_xpu.py TestCudaMultiGPU.test_mem_get_info This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

```

## Test Code Snippet

```python
def test_mem_get_info(self):
def _test(device: str | int | torch.device):
# Prevent PyTorch from reusing the allocated memory
torch.accelerator.empty_cache()
torch.accelerator.synchronize()
before_free_bytes, before_available_bytes = torch.get_device_module(
GPU_TYPE
).mem_get_info(device)
# increasing to 8MB to force acquiring a new block and overcome blocksize differences across platforms
t = torch.randn(1024 * 1024 * 8, device=device) # noqa: F841

if IS_JETSON or TEST_XPU:
# w/o syncing, mem_get_info will run before memory allocated has actually increased.
# This race condition causes consistent failure
torch.accelerator.synchronize()
after_free_bytes, after_available_bytes = torch.get_device_module(
GPU_TYPE
).mem_get_info(device)

self.assertLess(after_free_bytes, before_free_bytes)
self.assertEqual(before_available_bytes, after_available_bytes)

# Test calls with different device representations
_test(0)
_test(torch.device(GPU_TYPE))
_test(torch.device(f"{GPU_TYPE}:0"))
_test(GPU_TYPE)
_test(f"{GPU_TYPE}:0")
if TEST_MULTIGPU:
_test(1)
_test(torch.device(f"{GPU_TYPE}:1"))
_test(f"{GPU_TYPE}:1")

```

### Test location
test\xpu\test_cuda_multigpu_xpu.py
```python
def test_external_streams(self):
device = torch.get_device_module(GPU_TYPE).device(0)
with self._get_external_stream(device) as stream_v:
```
test case failed at

```
self.assertLess(after_free_bytes, before_free_bytes)
```

# TODO XPU PR
https://github.com/intel/torch-xpu-ops/pull/4294

### Versions

PyTorch: main 2.14.0a0
XPU Tests: torch-xpu-ops.test.xpu.test_cuda_multigpu_xpu.TestCudaMultiGPU.test_mem_get_info

Contributor guide

Open the contributing guide

Research direction

Start with test/xpu/test_cuda_multigpu_xpu.py and run PYTORCH_TEST_WITH_SLOW=1 python test/xpu/test_cuda_multigpu_xpu.py TestCudaMultiGPU.test_mem_get_info. Read the _test helper around the failing assert and the mem_get_info behavior before and after allocation. Done means the test reliably observes reduced free memory while the available-memory assertion still passes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.