NVIDIA / NVIDIA/cccl

MSVC C Parallel v2 HostJIT tests intermittently crash with access violations

Open
#10,802 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
2.5k
Forks
486
Avg merge
2d 6h
Merged PRs (30d)
295

Description

### Summary

The CTK 13.3/MSVC 14.50 C Parallel v2 HostJIT lane produced at least 17 transient SIGSEGVs between July 15 and August 12:

- `three_way_partition` (5):
[1](https://github.com/NVIDIA/cccl/actions/runs/29453202858/job/87480482826) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/29453202858/job/87506414471)),
[2](https://github.com/NVIDIA/cccl/actions/runs/29605630604/job/87968938456),
[3](https://github.com/NVIDIA/cccl/actions/runs/29835458273/job/88652098699),
[4](https://github.com/NVIDIA/cccl/actions/runs/30022728197/job/89259539724),
[5](https://github.com/NVIDIA/cccl/actions/runs/30669456229/job/91283909688)
- `unique_by_key` (4):
[1](https://github.com/NVIDIA/cccl/actions/runs/30505683738/job/90754919523) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/30505683738/job/90867040866)),
[2](https://github.com/NVIDIA/cccl/actions/runs/30608725773/job/91086771995),
[3](https://github.com/NVIDIA/cccl/actions/runs/31136885701/job/92738188619) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/31136885701/job/92885081675)),
[4](https://github.com/NVIDIA/cccl/actions/runs/31556964507/job/93991354758) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/31556964507/job/94098178483))
- `binary_search` (2):
[1](https://github.com/NVIDIA/cccl/actions/runs/29722003652/job/88286853663),
[2](https://github.com/NVIDIA/cccl/actions/runs/30808641566/job/91670183767) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/30808641566/job/91706521654))
- `segmented_reduce` (2):
[1](https://github.com/NVIDIA/cccl/actions/runs/29923398859/job/88934422862) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/29923398859/job/88976474588)),
[2](https://github.com/NVIDIA/cccl/actions/runs/30832248360/job/91748951807)
- `for` (2):
[1](https://github.com/NVIDIA/cccl/actions/runs/30290588131/job/90060595605),
[2](https://github.com/NVIDIA/cccl/actions/runs/31311782063/job/93240640338)
- `histogram` (1):
[1](https://github.com/NVIDIA/cccl/actions/runs/30813399127/job/91685425002) ([rerun passed](https://github.com/NVIDIA/cccl/actions/runs/30813399127/job/91787544687))
- `reduce` (1):
[1](https://github.com/NVIDIA/cccl/actions/runs/31604210969/job/94139039246)

The failures span unrelated PRs and multiple physical runners. At least seven unchanged reruns passed. Nodes associated with the latest failures also completed successful runs, arguing against a permanently broken runner.

During July 15–22 alone, 5 of 72 non-cancelled attempts failed this way (6.9%). Four additional matching crashes were observed during August 6–12.

### Analysis

The reported `cudaMalloc`, `cudaMemcpy`, and `nvrtcDestroyProgram` locations are not crash PCs. Catch2 reports the last completed assertion after a fatal SEH exception, producing `{Unknown expression after the reported line}`. In the inspected paths, the assertion counts place the fatal exception while the following [HostJIT build](https://github.com/NVIDIA/cccl/blob/ca40e942e0bc5b5fcfb52f3a516d53667549a83d/c/parallel/test/algorithm_execution.h#L130-L142) assertion is outstanding.

The working hypothesis is incomplete Windows stack protection. [PR #9605](https://github.com/NVIDIA/cccl/pull/9605) encountered the same signatures, including explicit stack overflows, and moved four Clang `ExecuteAction` calls onto an 8 MiB worker. LLVM linking, optimization/codegen, nvJitLink, and LLD still execute on the caller’s default 1 MiB stack.

No exercised pointer-lifetime, NVRTC buffer-lifetime, algorithm-specific, or permanently faulty-runner explanation accounts for the observed pattern. The current logs confirm a common transient failure but do not provide the actual crash PC needed to prove the stack hypothesis.

Contributor guide

Open the contributing guide

Research direction

Start with the HostJIT build assertions in c/parallel/test/algorithm_execution.h and compare the four Clang ExecuteAction calls moved in PR #9605. Reproduce the MSVC C Parallel v2 lane and capture the actual crash PC rather than relying on Catch2's reported assertion location. Done means identifying the common transient failure and documenting or implementing a justified fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
ci-cd, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.