llvm / llvm/llvm-project

[X86][GlobalISel] Legalizer crashes legalizing its own G_BUILD_VECTOR after scalarizing byte-lane G_MUL

Open
#216,655 1 comment 0 reactions 0 assignees View on GitHub
backend:X86 confirmed crash-on-valid llvm:globalisel
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

## Description

`llc -mtriple=x86_64-* -global-isel` aborts with "unable to legalize instruction" on an ordinary byte-lane vector multiply. `G_MUL` correctly resolves to `FewerElements`/scalarize for `<16 x i8>` (X86 has no packed byte-multiply instruction), but the `G_BUILD_VECTOR` the Legalizer itself creates to reassemble the scalarized lanes has no legality rule and the Legalizer crashes trying to legalize its own output.

## Reproducer

```llvm
define <16 x i8> @vecmul16(<16 x i8> %a, <16 x i8> %b) {
%r = mul <16 x i8> %a, %b
ret <16 x i8> %r
}
```

```
$ llc -mtriple=x86_64-unknown-linux-gnu -global-isel repro.ll -o -
LLVM ERROR: unable to legalize instruction: %2:_(<16 x s8>) = G_BUILD_VECTOR %35:_(s8), %36:_(s8), %37:_(s8), %38:_(s8), %39:_(s8), %40:_(s8), %41:_(s8), %42:_(s8), %43:_(s8), %44:_(s8), %45:_(s8), %46:_(s8), %47:_(s8), %48:_(s8), %49:_(s8), %50:_(s8) (in function: vecmul16)
```

Not specific to `<16 x i8>` — a smaller vector like `<4 x i8>` hits the same crash after argument-lowering widens it up to 16 lanes:

```llvm
define <4 x i8> @vecmul4(<4 x i8> %a, <4 x i8> %b) {
%r = mul <4 x i8> %a, %b
ret <4 x i8> %r
}
```
```
LLVM ERROR: unable to legalize instruction: %11:_(<16 x s8>) = G_BUILD_VECTOR ...
```

## Analysis

- Traced the rule chain in `X86LegalizerInfo.cpp`: `G_MUL`'s rules
(`legalFor`/`clampMinNumElements`/`clampMaxNumElements`) never mention
an `s8` element type at all (x86 has no packed byte-lane multiply), so
`scalarize(0)` — the catch-all `isVector` rule — fires for byte-element
vectors. That correctly produces `G_UNMERGE_VALUES` → 16x scalar
`G_MUL` → `G_BUILD_VECTOR`. But `G_BUILD_VECTOR` itself apparently has
no legality rule for a 16-lane `s8` destination, so the Legalizer's own
worklist, now processing the instruction it just inserted, aborts.
- I independently confirmed a correct instruction sequence for this exact
case *does* exist and *is* reachable: an experimental MLIR-based
instruction-selection frontend I'm working on
(`llc -enable-mlir-isel`, in an unrelated fork) produces the same
initial `G_UNMERGE_VALUES`/16x `G_MUL`/`G_BUILD_VECTOR` shape via a
different front-end path, and the *same* downstream
Legalizer/Combiner/RegBankSelect/InstructionSelect passes turn it into
correct code (the standard SSE2 byte-multiply idiom: widen each byte to
a word via `punpcklbw`/`punpckhbw`, `pmullw`, mask with `pand`, pack
back down with `packuswb`). So the target-specific pieces needed to
finish legalizing/selecting this shape exist and work when reached —
something about the Legalizer's own pass reaching `G_BUILD_VECTOR` from
*its own* scalarize action specifically hits the crash, rather than the
shape being fundamentally unsupported.
- With graceful fallback (`-global-isel-abort=0`), this silently and
correctly falls back to SelectionDAG for the affected function — no
user-visible failure. The hard abort above only reproduces under strict
abort-on-failure mode, which is `llc -global-isel`'s default (same
caveat as issue #216648).

Filing this as a heads-up for anyone working on X86 GlobalISel's vector
legalization, not as an urgent/user-facing issue — I don't believe this
affects any real user today given the point above.

## Environment

Reproduced at commit 513a02ee150335a84c39ab2e2d0f965f1b0e260e (llvm-project `main`). Confirmed via `git diff` that `llvm/lib/Target/X86/GISel/`, `llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp`, and `llvm/lib/CodeGen/GlobalISel/LegalizerInfo.cpp` are unmodified relative to that commit in the branch this was found on, so this should reproduce unmodified on upstream `main`.

Contributor guide

Open the contributing guide

Research direction

Start in llvm/lib/Target/X86/GISel/X86LegalizerInfo.cpp and trace the G_MUL scalarize rules alongside G_BUILD_VECTOR legality. Run the supplied vecmul16 and vecmul4 reproducer with llc -global-isel; done means the generated G_BUILD_VECTOR is legalized without an abort and the existing downstream selection produces correct code.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.