pytorch / pytorch/pytorch

[Test] `test_linalg.py` failures when CUDA profiling runs after CPU-only profiling

Open
#197,005 1 comment 0 reactions 0 assignees View on GitHub
bot-triaged oncall: profiler
Dominant language
Python
Stars
103k
Forks
29.5k
PR merge metrics
PR metrics pending

Description

### 🐛 Describe the bug

I'm seeing failures in `test_call_count_tunableop_cuda_float32` due to other tests (for example, `test_matmul_folds_viewable`) registering Kineto with CPU-only profiling before this test gets called. The profiler therefore does not see the three CUDA kernels that run and reports 0. This problem has appeared before in `test_profiler.py` and was fixed by #186970, which registers both CPU and CUDA profiler activity during setup. The same fix works for `test_linalg.py`.

Given that this problem is now appearing in multiple places, a few questions:
1. Does it make sense to reconsider how Kineto registration works? From a different perspective, is calling the profiler multiple times in the same process with different profiler activities considered a supported use case?
2. Should the profiler notify users somehow if the requested profiling modes are not supported by the existing registration?
3. Is the test-side solution implemented by #186970 the "blessed" path for now? If so, the quickest fix would be to move the Kineto setup to one of the test util files and import/use it both in `test_profiler.py` and `test_linalg.py`.

The `test_call_count_tunableop_cuda_float32` failure:
```
======================================================================
FAIL: test_call_count_tunableop_cuda_float32 (__main__.TestLinalgCudaOnlyCUDA.test_call_count_tunableop_cuda_float32)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/agents-local/workspace/pytorch/torch/testing/_internal/common_utils.py", line 3905, in wrapper
method(*args, **kwargs)
File "/tmp/agents-local/workspace/pytorch/torch/testing/_internal/common_utils.py", line 3905, in wrapper
method(*args, **kwargs)
File "/tmp/agents-local/workspace/pytorch/torch/testing/_internal/common_device_type.py", line 672, in instantiated_test
result = test(self, **param_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/agents-local/workspace/pytorch/test/test_linalg.py", line 11519, in test_call_count_tunableop
self.assertEqual(kernel_count, 3)
File "/tmp/agents-local/workspace/pytorch/torch/testing/_internal/common_utils.py", line 4955, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Scalars are not equal!

Expected 3 but got 0.
Absolute difference: 3
Relative difference: 1.0

To execute this test, run the following from the base repo dir:
python test/test_linalg.py TestLinalgCudaOnlyCUDA.test_call_count_tunableop_cuda_float32

This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0

----------------------------------------------------------------------
```

### Versions

Latest main on GB200

cc @robieta @chaekit @guotuofeng @guyang3532 @dzhulgakov @davidberard98 @briancoutinho @sraikund16 @sanrise @mwootton @divyanshk @jiannanWang @scotts @ryanzhang22

Contributor guide

Open the contributing guide

Research direction

Start with test/test_linalg.py and the test-side Kineto setup added for test_profiler.py in #186970. Compare how profiler activities are registered before test_call_count_tunableop_cuda_float32 runs, then run the focused command from the failure report. Done means the test observes all three CUDA kernels even after CPU-only profiling tests have run.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
performance, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.