[Test] `test_linalg.py` failures when CUDA profiling runs after CPU-only profiling
- Dominant language
- Python
- Stars
- 103k
- Forks
- 29.5k
- PR merge metrics
- PR metrics pending
Description
### 🐛 Describe the bug
I'm seeing failures in `test_call_count_tunableop_cuda_float32` due to other tests (for example, `test_matmul_folds_viewable`) registering Kineto with CPU-only profiling before this test gets called. The profiler therefore does not see the three CUDA kernels that run and reports 0. This problem has appeared before in `test_profiler.py` and was fixed by #186970, which registers both CPU and CUDA profiler activity during setup. The same fix works for `test_linalg.py`.
Given that this problem is now appearing in multiple places, a few questions:
1. Does it make sense to reconsider how Kineto registration works? From a different perspective, is calling the profiler multiple times in the same process with different profiler activities considered a supported use case?
2. Should the profiler notify users somehow if the requested profiling modes are not supported by the existing registration?
3. Is the test-side solution implemented by #186970 the "blessed" path for now? If so, the quickest fix would be to move the Kineto setup to one of the test util files and import/use it both in `test_profiler.py` and `test_linalg.py`.
The `test_call_count_tunableop_cuda_float32` failure:
```
======================================================================
FAIL: test_call_count_tunableop_cuda_float32 (__main__.TestLinalgCudaOnlyCUDA.test_call_count_tunableop_cuda_float32)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/tmp/agents-local/workspace/pytorch/torch/testing/_internal/common_utils.py", line 3905, in wrapper
method(*args, **kwargs)
File "/tmp/agents-local/workspace/pytorch/torch/testing/_internal/common_utils.py", line 3905, in wrapper
method(*args, **kwargs)
File "/tmp/agents-local/workspace/pytorch/torch/testing/_internal/common_device_type.py", line 672, in instantiated_test
result = test(self, **param_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/tmp/agents-local/workspace/pytorch/test/test_linalg.py", line 11519, in test_call_count_tunableop
self.assertEqual(kernel_count, 3)
File "/tmp/agents-local/workspace/pytorch/torch/testing/_internal/common_utils.py", line 4955, in assertEqual
raise error_metas.pop()[0].to_error( # type: ignore[index]
AssertionError: Scalars are not equal!
Expected 3 but got 0.
Absolute difference: 3
Relative difference: 1.0
To execute this test, run the following from the base repo dir:
python test/test_linalg.py TestLinalgCudaOnlyCUDA.test_call_count_tunableop_cuda_float32
This message can be suppressed by setting PYTORCH_PRINT_REPRO_ON_FAILURE=0
----------------------------------------------------------------------
```
### Versions
Latest main on GB200
cc @robieta @chaekit @guotuofeng @guyang3532 @dzhulgakov @davidberard98 @briancoutinho @sraikund16 @sanrise @mwootton @divyanshk @jiannanWang @scotts @ryanzhang22
Contributor guide
Research direction
Start with test/test_linalg.py and the test-side Kineto setup added for test_profiler.py in #186970. Compare how profiler activities are registered before test_call_count_tunableop_cuda_float32 runs, then run the focused command from the failure report. Done means the test observes all three CUDA kernels even after CPU-only profiling tests have run.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- performance, testing-qa
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100