microsoft / microsoft/DirectXShaderCompiler

DXCompiler segfaults when it's impossible to allocate clip/cull registers

Open
#6,964 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug crash diagnostic incorrect-code
Dominant language
C++
Stars
3.7k
Forks
900
Avg merge
2d 11h
Merged PRs (30d)
44

Description

Description
DXCompiler segfaults when trying to compile shaders in which it's impossible to allocate clip/cull registers.

Steps to Reproduce
Compile this shader with DXC (profile vs_6_0):

struct vs_out
{
    float4 position : SV_Position;
    float clip[3] : SV_ClipDistance;
};

void main(uint id : SV_VertexID, out vs_out o)
{
    o.position = float4(0.0f, 0.0f, 0.0f, 1.0f);
    o.clip[0] = 1.0f;
    o.clip[1] = 1.0f;
    o.clip[2] = 1.0f;
}

The compiler will segfault.

Also see on Compiler Explorer.

Other shaders will cause the same behavior:

struct vs_out
{
    float4 position : SV_Position;
    float clip[2] : SV_ClipDistance1;
};

void main(uint id : SV_VertexID, out vs_out o)
{
    o.position = float4(0.0f, 0.0f, 0.0f, 1.0f);
    o.clip[0] = 1.0f;
    o.clip[1] = 1.0f;
}
static const float2 vertices[3] =
{
    {-1.0f,  1.0f},
    { 3.0f,  1.0f},
    {-1.0f, -3.0f},
};

struct vs_out
{
    float4 position : SV_Position;
    float2 clip : SV_ClipDistance;
    float3 clip2 : SV_ClipDistance1;
    float3 cull : SV_CullDistance;
};

void main(uint id : SV_VertexID, out vs_out o)
{
    const float2 pos = vertices[id];
    o.position = float4(pos, 0.0f, 1.0f);
    o.clip = 1.0f;
    o.clip2 = 1.0f;
    o.cull = 1.0f;
}
static const float2 vertices[3] =
{
    {-1.0f,  1.0f},
    { 3.0f,  1.0f},
    {-1.0f, -3.0f},
};

struct vs_out
{
    float4 position : SV_Position;
    float2 clip : SV_ClipDistance;
    float3 clip2 : SV_ClipDistance1;
    float1 cull : SV_CullDistance;
    float2 cull2 : SV_CullDistance1;
};

void main(uint id : SV_VertexID, out vs_out o)
{
    const float2 pos = vertices[id];
    o.position = float4(pos, 0.0f, 1.0f);
    o.clip = 1.0f;
    o.clip2 = 1.0f;
    o.cull = 1.0f;
    o.cull2 = 1.0f;
}

Actual Behavior
The shader is indeed invalid (you cannot allocate three registers for clip/cull distances), so I expect an error message, but not a segfault. The same applies to the other shaders, except that they fail for different reasons (all related to allocating clip/cull registers).

Environment

  • DXC version: libdxcompiler.so: 1.8(dev;4662-416fab6b); libdxil.so: 1.8
  • Host Operating System: Debian unstable

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by compiling the provided HLSL shaders with DXC using the vs_6_0 profile and confirm the clip/cull register-allocation failure. Trace the compiler path that handles these allocations and replace the segfault with an error diagnostic; done means all listed invalid shaders fail without crashing.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.