[X86][GlobalISel] Legalizer crashes legalizing its own G_BUILD_VECTOR after scalarizing byte-lane G_MUL
- Dominant language
- LLVM
- Stars
- 40.5k
- Forks
- 18.7k
- PR merge metrics
- PR metrics pending
Description
## Description
`llc -mtriple=x86_64-* -global-isel` aborts with "unable to legalize instruction" on an ordinary byte-lane vector multiply. `G_MUL` correctly resolves to `FewerElements`/scalarize for `<16 x i8>` (X86 has no packed byte-multiply instruction), but the `G_BUILD_VECTOR` the Legalizer itself creates to reassemble the scalarized lanes has no legality rule and the Legalizer crashes trying to legalize its own output.
## Reproducer
```llvm
define <16 x i8> @vecmul16(<16 x i8> %a, <16 x i8> %b) {
%r = mul <16 x i8> %a, %b
ret <16 x i8> %r
}
```
```
$ llc -mtriple=x86_64-unknown-linux-gnu -global-isel repro.ll -o -
LLVM ERROR: unable to legalize instruction: %2:_(<16 x s8>) = G_BUILD_VECTOR %35:_(s8), %36:_(s8), %37:_(s8), %38:_(s8), %39:_(s8), %40:_(s8), %41:_(s8), %42:_(s8), %43:_(s8), %44:_(s8), %45:_(s8), %46:_(s8), %47:_(s8), %48:_(s8), %49:_(s8), %50:_(s8) (in function: vecmul16)
```
Not specific to `<16 x i8>` — a smaller vector like `<4 x i8>` hits the same crash after argument-lowering widens it up to 16 lanes:
```llvm
define <4 x i8> @vecmul4(<4 x i8> %a, <4 x i8> %b) {
%r = mul <4 x i8> %a, %b
ret <4 x i8> %r
}
```
```
LLVM ERROR: unable to legalize instruction: %11:_(<16 x s8>) = G_BUILD_VECTOR ...
```
## Analysis
- Traced the rule chain in `X86LegalizerInfo.cpp`: `G_MUL`'s rules
(`legalFor`/`clampMinNumElements`/`clampMaxNumElements`) never mention
an `s8` element type at all (x86 has no packed byte-lane multiply), so
`scalarize(0)` — the catch-all `isVector` rule — fires for byte-element
vectors. That correctly produces `G_UNMERGE_VALUES` → 16x scalar
`G_MUL` → `G_BUILD_VECTOR`. But `G_BUILD_VECTOR` itself apparently has
no legality rule for a 16-lane `s8` destination, so the Legalizer's own
worklist, now processing the instruction it just inserted, aborts.
- I independently confirmed a correct instruction sequence for this exact
case *does* exist and *is* reachable: an experimental MLIR-based
instruction-selection frontend I'm working on
(`llc -enable-mlir-isel`, in an unrelated fork) produces the same
initial `G_UNMERGE_VALUES`/16x `G_MUL`/`G_BUILD_VECTOR` shape via a
different front-end path, and the *same* downstream
Legalizer/Combiner/RegBankSelect/InstructionSelect passes turn it into
correct code (the standard SSE2 byte-multiply idiom: widen each byte to
a word via `punpcklbw`/`punpckhbw`, `pmullw`, mask with `pand`, pack
back down with `packuswb`). So the target-specific pieces needed to
finish legalizing/selecting this shape exist and work when reached —
something about the Legalizer's own pass reaching `G_BUILD_VECTOR` from
*its own* scalarize action specifically hits the crash, rather than the
shape being fundamentally unsupported.
- With graceful fallback (`-global-isel-abort=0`), this silently and
correctly falls back to SelectionDAG for the affected function — no
user-visible failure. The hard abort above only reproduces under strict
abort-on-failure mode, which is `llc -global-isel`'s default (same
caveat as issue #216648).
Filing this as a heads-up for anyone working on X86 GlobalISel's vector
legalization, not as an urgent/user-facing issue — I don't believe this
affects any real user today given the point above.
## Environment
Reproduced at commit 513a02ee150335a84c39ab2e2d0f965f1b0e260e (llvm-project `main`). Confirmed via `git diff` that `llvm/lib/Target/X86/GISel/`, `llvm/lib/CodeGen/GlobalISel/LegalizerHelper.cpp`, and `llvm/lib/CodeGen/GlobalISel/LegalizerInfo.cpp` are unmodified relative to that commit in the branch this was found on, so this should reproduce unmodified on upstream `main`.
Contributor guide
Research direction
Start in llvm/lib/Target/X86/GISel/X86LegalizerInfo.cpp and trace the G_MUL scalarize rules alongside G_BUILD_VECTOR legality. Run the supplied vecmul16 and vecmul4 reproducer with llc -global-isel; done means the generated G_BUILD_VECTOR is legalized without an abort and the existing downstream selection produces correct code.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100