[BUG]: tileiras SIGSEGV when occupancy=2 is requested for a 32-wide fused tile kernel

Abierto
#96 1 comentario 0 reacciones 0 asignados Ver en GitHub

Nadie ha tomado este issue todavía.

Evaluación

Dificultad
4/5
Tiempo estimado
3-5 días
Aptitud para principiantes
52/100
Tipo de issue
Error
Claridad
Bastante claro
Estado de actividad
Activo
Stack tecnológico
python
Área
compilers

Línea de trabajo

Comienza ejecutando repro.py en un proceso nuevo con la configuración de caché y volcado de errores indicada; después, compara occupancy=2 con los controles que pasan. Rastrea la invocación de tileiras que devuelve SIGSEGV y utiliza las variantes indicadas de occupancy, worker-warp, block-size y allocation para aislar el fallo del compilador. Se considera terminado cuando el reproductor ya no se bloquea y una prueba de regresión cubre la configuración que falla.

Escrito por el modelo de indexación a partir del texto del issue.

Descripción

bug
cuTile Python version

1.5.0. The same reproducer also fails with 1.4.0.

CUDA Toolkit version

13.3 (tileiras V13.3.36)

Which installation method does this occur on?

Pip

Describe the bug

tileiras terminates with SIGSEGV when occupancy=2 is requested for the
kernel below. Automatic occupancy and occupancy=1 compile. The crash happens
during compilation, before the intentionally small input can execute.

I expected the occupancy request either to compile, to be treated as a hint that
cannot be met, or to produce a clear resource diagnostic. A scheduling/resource
request should not terminate the native compiler.

This was reduced from two Cholesky _left_superpanel failures. Their source
inputs were float32[1,4096,4096]; the compiler-only reproducer needs one
float32[1,4,4] tensor while preserving the 32x32, 32x64, and 64x64 compile-time
tiles.

Minimum reproducible example
import torch
import cuda.tile as ct


ConstInt = ct.Constant[int]
ZERO = ct.PaddingMode.ZERO


def factor(a, block: ConstInt):
    cols = ct.arange(block, dtype=ct.int32)[None, :]
    for p in range(block):
        pivot = ct.extract(a, (p, p), shape=(1, 1))
        column = ct.extract(a, (0, p), shape=(block, 1)) / pivot
        outer = column * column.transpose(0, 1)
        a = ct.where(cols > p, a - outer, a)
    return a


def solve(panel, diagonal, block: ConstInt):
    cols = ct.arange(block, dtype=ct.int32)[None, :]
    for p in range(block):
        pivot = ct.extract(diagonal, (p, p), shape=(1, 1))
        solved = ct.extract(panel, (0, p), shape=(block, 1)) / pivot
        column = ct.extract(diagonal, (0, p), shape=(block, 1))
        panel = ct.where(
            cols > p,
            panel - solved * column.transpose(0, 1),
            panel,
        )
    return panel


@ct.kernel(opt_level=2, occupancy=2)
def kernel(a, step, block: ConstInt):
    rows = block
    width = 2 * block
    next_step = step + 1

    work = ct.load(a, (0, 0, 0), shape=(1, rows, width), padding_mode=ZERO)
    work = work.reshape((rows, width))
    cross = ct.load(a, (0, 1, 0), shape=(1, block, block), padding_mode=ZERO)
    cross = cross.reshape((block, block))
    diagonal = ct.load(a, (0, 1, 1), shape=(1, block, block), padding_mode=ZERO)
    diagonal = diagonal.reshape((block, block))

    for prior in range(step):
        left = ct.load(a, (0, 0, prior), shape=(1, rows, width), padding_mode=ZERO)
        right = ct.load(a, (0, 0, prior), shape=(1, width, width), padding_mode=ZERO)
        left = left.reshape((rows, width))
        right = right.reshape((width, width))
        work = ct.mma(
            left.astype(ct.tfloat32),
            (-right.transpose(0, 1)).astype(ct.tfloat32),
            work,
        )
        diagonal = ct.mma(
            left.astype(ct.tfloat32),
            (-left.transpose(0, 1)).astype(ct.tfloat32),
            diagonal,
        )

    first_panel = ct.extract(work, (0, 0), shape=(rows, block))
    second_panel = ct.mma(
        first_panel.astype(ct.tfloat32),
        (-cross.transpose(0, 1)).astype(ct.tfloat32),
        first_panel,
    )
    second_panel = solve(second_panel, diagonal, block)

    ct.store(a, (0, 0, 0), first_panel.reshape((1, rows, block)))
    if ct.bid(1) == 0:
        ct.store(a, (0, 1, 1), diagonal.reshape((1, block, block)))

    if ct.bid(1) == next_step:
        next_diagonal = ct.load(
            a,
            (0, next_step, next_step),
            shape=(1, block, block),
            padding_mode=ZERO,
        ).reshape((block, block))
        next_diagonal = ct.mma(
            second_panel.astype(ct.tfloat32),
            (-second_panel.transpose(0, 1)).astype(ct.tfloat32),
            next_diagonal,
        )
        next_diagonal = factor(next_diagonal, block)
        ct.store(
            a,
            (0, next_step, next_step),
            next_diagonal.reshape((1, block, block)),
        )


a = torch.empty((1, 4, 4), device="cuda", dtype=torch.float32)
ct.launch(torch.cuda.current_stream(), (1, 1), kernel, (a, 0, 32))

Run it in a fresh process and compiler cache. Crash dumps are disabled only to
avoid the separate masking problem in #92.

run=$(mktemp -d)
CUDA_TILE_CACHE_DIR=off \
CUDA_TILE_TEMP_DIR="$run" \
CUDA_TILE_ENABLE_CRASH_DUMP=0 \
python repro.py
Relevant log output
subprocess.CalledProcessError: Command '['/usr/local/cuda/bin/tileiras',
  '/tmp/.../kernel....bytecode', '-o',
  '/tmp/.../kernel....cubin', '--gpu-name', 'sm_120',
  '-O2', '--lineinfo']' died with <Signals.SIGSEGV: 11>.

cuda.tile._exception.TileCompilerExecutionError: Return code -11
Unknown location

The corresponding direct tileiras invocation exits 139 and emits no cubin.
Two fresh 1.4.0 processes produced identical failing bytecode; a fresh 1.5.0
process also produced a failing compiler input.

Environment
OS: Ubuntu 22.04.5 LTS, Linux 6.8.0-90-generic x86_64
GPU: NVIDIA RTX PRO 6000 Blackwell Server Edition, compute capability 12.0
Driver: 580.126.09
CUDA toolkit: 13.3; nvcc 13.3.33; tileiras V13.3.36
Python: 3.13.14
PyTorch: 2.12.0+cu130 (bundled CUDA runtime 13.0)
cuTile Python: 1.5.0; also reproduced on 1.4.0
CPU: AMD EPYC 9355, 16 vCPUs
Other details

The two original failing source variants were:

  • occupancy=2;
  • occupancy=2, num_worker_warps=4.

Both original inputs compiled the same _left_superpanel body and failed at
n=4096. The reduced controls isolate the request:

  • automatic occupancy passes;
  • occupancy=1 passes;
  • occupancy=2, num_worker_warps=4 has the same crash;
  • occupancy=2, num_worker_warps=8 passes at this reduced boundary;
  • occupancy=2 with block=16 passes;
  • shrinking the allocated tensor from 4x4 to 3x3 changes its alignment signature
    and passes.

The original no-worker-warp and four-worker-warp compiler inputs differ only by
worker-warp metadata and both exit 139. This makes the occupancy-two request the
common trigger for this compact configuration.

Contributing Guidelines
  • I agree to follow cuTile Python's contributing guidelines
  • I searched the open bugs and found no duplicate for this report
Lenguaje dominante
Python
Estrellas
2.2k
Forks
155
Métricas de merge de PR
Sin PR fusionados en 30 d

Guía de contribución

Abrir la guía de contribución

Primeros pasos

  1. Lee el issue completo y luego la guía de contribución del proyecto.
  2. Comenta en el issue que vas a ocuparte — evita que dos personas hagan lo mismo.
  3. Haz un fork del repositorio y trabaja en una rama.
  4. Abre un pull request que haga referencia al número del issue.

Más de NVIDIA/cutile-python

Todos los issues de NVIDIA/cutile-python

Issues similares

Más issues de Python

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.