microsoft / microsoft/DirectXShaderCompiler
DXCompiler segfaults when it's impossible to allocate clip/cull registers
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 3.7k
- Forks
- 900
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 44
Description
Description
DXCompiler segfaults when trying to compile shaders in which it's impossible to allocate clip/cull registers.
Steps to Reproduce
Compile this shader with DXC (profile vs_6_0):
struct vs_out
{
float4 position : SV_Position;
float clip[3] : SV_ClipDistance;
};
void main(uint id : SV_VertexID, out vs_out o)
{
o.position = float4(0.0f, 0.0f, 0.0f, 1.0f);
o.clip[0] = 1.0f;
o.clip[1] = 1.0f;
o.clip[2] = 1.0f;
}
The compiler will segfault.
Also see on Compiler Explorer.
Other shaders will cause the same behavior:
struct vs_out
{
float4 position : SV_Position;
float clip[2] : SV_ClipDistance1;
};
void main(uint id : SV_VertexID, out vs_out o)
{
o.position = float4(0.0f, 0.0f, 0.0f, 1.0f);
o.clip[0] = 1.0f;
o.clip[1] = 1.0f;
}
static const float2 vertices[3] =
{
{-1.0f, 1.0f},
{ 3.0f, 1.0f},
{-1.0f, -3.0f},
};
struct vs_out
{
float4 position : SV_Position;
float2 clip : SV_ClipDistance;
float3 clip2 : SV_ClipDistance1;
float3 cull : SV_CullDistance;
};
void main(uint id : SV_VertexID, out vs_out o)
{
const float2 pos = vertices[id];
o.position = float4(pos, 0.0f, 1.0f);
o.clip = 1.0f;
o.clip2 = 1.0f;
o.cull = 1.0f;
}
static const float2 vertices[3] =
{
{-1.0f, 1.0f},
{ 3.0f, 1.0f},
{-1.0f, -3.0f},
};
struct vs_out
{
float4 position : SV_Position;
float2 clip : SV_ClipDistance;
float3 clip2 : SV_ClipDistance1;
float1 cull : SV_CullDistance;
float2 cull2 : SV_CullDistance1;
};
void main(uint id : SV_VertexID, out vs_out o)
{
const float2 pos = vertices[id];
o.position = float4(pos, 0.0f, 1.0f);
o.clip = 1.0f;
o.clip2 = 1.0f;
o.cull = 1.0f;
o.cull2 = 1.0f;
}
Actual Behavior
The shader is indeed invalid (you cannot allocate three registers for clip/cull distances), so I expect an error message, but not a segfault. The same applies to the other shaders, except that they fail for different reasons (all related to allocating clip/cull registers).
Environment
- DXC version: libdxcompiler.so: 1.8(dev;4662-416fab6b); libdxil.so: 1.8
- Host Operating System: Debian unstable
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by compiling the provided HLSL shaders with DXC using the vs_6_0 profile and confirm the clip/cull register-allocation failure. Trace the compiler path that handles these allocations and replace the segfault with an error diagnostic; done means all listed invalid shaders fail without crashing.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100