MSVC C Parallel v2 HostJIT tests intermittently crash with access violations
- Dominant language
- C++
- Stars
- 2.5k
- Forks
- 486
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 295
Description
### Summary
The CTK 13.3/MSVC 14.50 C Parallel v2 HostJIT lane produced at least 17 transient SIGSEGVs between July 15 and August 12:
- `three_way_partition` (5):
[1](https://github.com/NVIDIA/cccl/actions/runs/29453202858/job/87480482826) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/29453202858/job/87506414471)),
[2](https://github.com/NVIDIA/cccl/actions/runs/29605630604/job/87968938456),
[3](https://github.com/NVIDIA/cccl/actions/runs/29835458273/job/88652098699),
[4](https://github.com/NVIDIA/cccl/actions/runs/30022728197/job/89259539724),
[5](https://github.com/NVIDIA/cccl/actions/runs/30669456229/job/91283909688)
- `unique_by_key` (4):
[1](https://github.com/NVIDIA/cccl/actions/runs/30505683738/job/90754919523) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/30505683738/job/90867040866)),
[2](https://github.com/NVIDIA/cccl/actions/runs/30608725773/job/91086771995),
[3](https://github.com/NVIDIA/cccl/actions/runs/31136885701/job/92738188619) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/31136885701/job/92885081675)),
[4](https://github.com/NVIDIA/cccl/actions/runs/31556964507/job/93991354758) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/31556964507/job/94098178483))
- `binary_search` (2):
[1](https://github.com/NVIDIA/cccl/actions/runs/29722003652/job/88286853663),
[2](https://github.com/NVIDIA/cccl/actions/runs/30808641566/job/91670183767) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/30808641566/job/91706521654))
- `segmented_reduce` (2):
[1](https://github.com/NVIDIA/cccl/actions/runs/29923398859/job/88934422862) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/29923398859/job/88976474588)),
[2](https://github.com/NVIDIA/cccl/actions/runs/30832248360/job/91748951807)
- `for` (2):
[1](https://github.com/NVIDIA/cccl/actions/runs/30290588131/job/90060595605),
[2](https://github.com/NVIDIA/cccl/actions/runs/31311782063/job/93240640338)
- `histogram` (1):
[1](https://github.com/NVIDIA/cccl/actions/runs/30813399127/job/91685425002) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/30813399127/job/91787544687))
- `reduce` (1):
[1](https://github.com/NVIDIA/cccl/actions/runs/31604210969/job/94139039246)
The failures span unrelated PRs and multiple physical runners. At least seven unchanged reruns passed. Nodes associated with the latest failures also completed successful runs, arguing against a permanently broken runner.
During July 15–22 alone, 5 of 72 non-cancelled attempts failed this way (6.9%). Four additional matching crashes were observed during August 6–12.
### Analysis
The reported `cudaMalloc`, `cudaMemcpy`, and `nvrtcDestroyProgram` locations are not crash PCs. Catch2 reports the last completed assertion after a fatal SEH exception, producing `{Unknown expression after the reported line}`. In the inspected paths, the assertion counts place the fatal exception while the following [HostJIT build](https://github.com/NVIDIA/cccl/blob/ca40e942e0bc5b5fcfb52f3a516d53667549a83d/c/parallel/test/algorithm_execution.h#L130-L142) assertion is outstanding.
The working hypothesis is incomplete Windows stack protection. [PR #9605](https://github.com/NVIDIA/cccl/pull/9605) encountered the same signatures, including explicit stack overflows, and moved four Clang `ExecuteAction` calls onto an 8 MiB worker. LLVM linking, optimization/codegen, nvJitLink, and LLD still execute on the caller’s default 1 MiB stack.
No exercised pointer-lifetime, NVRTC buffer-lifetime, algorithm-specific, or permanently faulty-runner explanation accounts for the observed pattern. The current logs confirm a common transient failure but do not provide the actual crash PC needed to prove the stack hypothesis.
Contributor guide
Research direction
Start with the HostJIT build assertions in c/parallel/test/algorithm_execution.h and compare the four Clang ExecuteAction calls moved in PR #9605. Reproduce the MSVC C Parallel v2 lane and capture the actual crash PC rather than relying on Catch2's reported assertion location. Done means identifying the common transient failure and documenting or implementing a justified fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- ci-cd, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100