NVIDIA / NVIDIA/cutlass

Validation gaps: grouped/split-K can_implement accept crashing configs; batch grid wraps at 65536; Sm80 EVT mode gates are assert-only

Open
#3,542 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

CUTLASS C++
Dominant language
C++
Stars
10.5k
Forks
2.1k
Avg merge
3d 11h
Merged PRs (30d)
7

Description

Description

Several can_implement / parameter-validation paths accept configurations that cannot execute, converting what should be a clean kErrorInvalidProblem into a crash, an unusable launch, or silent garbage in release builds.

  1. Grouped GEMM accepts a config it then crashes on - gemm/kernel/gemm_grouped.h:287-294 returns kSuccess unconditionally from both can_implement overloads (no alignment checks, no host_problem_sizes check), while device/base_grouped.h:186-189 turns a null host_problem_sizes into a -1 tile count and initialize() (base_grouped.h:362-366) feeds that into precompute, which dereferences the null array. With a scheduler that sets kRequiresPrecomputation (e.g. the host-precompute grouped scheduler), can_implement succeeds and initialize segfaults on the host.

  2. gemm_splitk_parallel divides by zero after unconditional success - device/gemm_splitk_parallel.h:252-254 returns kSuccess unconditionally; with split_k_slices == 0 the kernel Params ctor computes problem_size.k() / grid_tiled_shape.k() (kernel/gemm_splitk_parallel.h:113) and the host faults on integer division by zero. Sibling wrappers validate the slice count; this one validates nothing.

  3. Batch counts that are multiples of 65536 produce gridDim.z == 0 - threadblock/threadblock_swizzle.h:255-257 computes the batched grid z as

    batch_count % (1 << 16)
    

    which is 0 for exactly the multiples of the limit instead of clamping to it. Consumed by the batched/array grid shapes (device/gemm_batched.h, device/gemm_array.h) and by device/gemv.h:107-110 / gemv_blockscaled.h:110-113. CUDA rejects the zero-dimension launch and the wrapper reports kErrorInternal even though can_implement accepted the arguments. The grid-stride loops inside these kernels would handle >65535 batches fine if the launch dimension were clamped rather than wrapped.

  4. Sm80 EVT kernel gates unsupported modes behind plain assert() - gemm/kernel/gemm_universal_with_visitor.h:149-154 rejects kGemmSplitKParallel, split-K-serial GEMM, and kArray modes with bare assert(...). In release builds (NDEBUG) those vanish and the kernel runs with garbage semantics. Other kernels return Status::kErrorNotSupported from can_implement; this one defers to a debug-only assertion.

Suggested fix

Return kErrorInvalidProblem / kErrorNotSupported from the corresponding can_implement paths for (1), (2), and (4), and clamp rather than wrap the batch grid dimension in (3).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the can_implement paths in gemm_grouped.h, device/gemm_splitk_parallel.h, and gemm_universal_with_visitor.h, then trace the related initialization and launch calculations in base_grouped.h, kernel/gemm_splitk_parallel.h, and threadblock_swizzle.h. Check the batched grid consumers in device/gemm_batched.h, device/gemm_array.h, device/gemv.h, and gemv_blockscaled.h. Done means invalid configurations return the documented error statuses and batch dimensions no longer produce zero-sized launches.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.