microsoft / microsoft/DirectXShaderCompiler

[DXIL] Crash when a function output parameter of a matrix type requires a cast to another matrix type of equal dimensions (e.g., float3x3 -> half3x3)

Open
#8,809 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug needs-triage
Dominant language
C++
Stars
3.7k
Forks
900
Avg merge
2d 11h
Merged PRs (30d)
44

Description

Description
The following code will cause a crash when compiling for DXIL output:

void crash_test(out float3x3 output)
{
    output = (float3x3)0;
}

float4 main() : SV_Position
{
    half3x3 output;
    crash_test(output); // <- the output parameter is not able to cast between halfNxM and floatNxM
    return (float4)0;
}

The cause is the function crash_test() having an output parameter of type float3x3, but passing in a variable of type half3x3 when calling it from main(). After a bit of digging in a debug build, it seem a DXASSERT is being hit in ConvertScalarOrVector() because the source and destination values are neither scalars nor vectors. It is getting to that code because there is no specific behavior defined in CGMSHLSLRuntime::EmitHLSLOutParamConversionCopyBack() for handling matrix type casting.

Steps to Reproduce

Actual Behavior
The following DXASSERT in ConvertScalarOrVector() is being hit when attempting to compile the above code for DXIL:

  DXASSERT(
      (SrcTy->isIntOrIntVectorTy() || SrcTy->isFPOrFPVectorTy()) &&
          (DstTy->isIntOrIntVectorTy() || DstTy->isFPOrFPVectorTy()),
      "EmitNumericConversion can only be used with int/float scalars/vectors.");

Environment

  • DXC version: dxcompiler.dll: 1.10(5434-75cab590)(1.9.0.15434) - 1.9.0.15434 (main, 75cab590f)
  • Host Operating System: Windows 11

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing CGMSHLSLRuntime::EmitHLSLOutParamConversionCopyBack() and the failing path into ConvertScalarOrVector(). Reproduce the crash by compiling the HLSL example with -E main -T vs_6_0, then verify that equal-dimension matrix casts for output parameters compile without triggering the assertion.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.