obsproject / obsproject/obs-studio

(apparent) Shader pixelshader parser/compiler bug.

Open
#5,626 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

issue/confirmed platform/windows
Dominant language
C
Stars
76.4k
Forks
10.2k
Avg merge
4d 23h
Merged PRs (30d)
12

Description

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

27.1.3

OBS Studio Version (Other)

No response

OBS Studio Log URL

https://obsproject.com/logs/Me2NHJSdFZ7a3TRo

OBS Studio Crash Log URL

No response

Expected Behavior

pixel shader should output the specified sampled pixel, regardless of what is computed/sampled in the function.

Current Behavior

While writing a simple custom shader for the obs-multisource-effect plugin, I seem to have hit a shader parser/compiler bug. I'm not familiar with OBS' shader system, but I'm a long-time D3D shader effect programmer so this is clearly wrong:

Using obs-multisource-effect's 'add.effect' example - this just adds together two texture inputs, and works correctly:

uniform float4x4 ViewProj;
uniform texture2d src0;
uniform texture2d src1;

sampler_state def_sampler {
	Filter   = Linear;
	AddressU = Clamp;
	AddressV = Clamp;
};

struct VertInOut {
	float4 pos : POSITION;
	float2 uv  : TEXCOORD0;
};

VertInOut VSDefault(VertInOut vert_in)
{
	VertInOut vert_out;
	vert_out.pos = mul(float4(vert_in.pos.xyz, 1.0), ViewProj);
	vert_out.uv  = vert_in.uv;
	return vert_out;
}

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
	float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
	float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
	return rgba0 + rgba1;
}

technique Draw
{
	pass
	{
		vertex_shader = VSDefault(vert_in);
		pixel_shader  = PSDrawBare(vert_in);
	}
}

To demo the bug: when outputting only the src0 texture like this, it works:

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
	float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
	float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
	return rgba0;
}

However, trying to output the src1 texture gives the src0 texture instead (this is wrong):

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
	float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
	float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
	return rgba1;
}

It only works correctly if you remove/comment out the src0 sampler line (should not be required):

float4 PSDrawBare(VertInOut vert_in) : TARGET
{
//	float4 rgba0 = src0.Sample(def_sampler, vert_in.uv);
	float4 rgba1 = src1.Sample(def_sampler, vert_in.uv);
	return rgba1;
}

I hit this bug (without a workaround) in a similar simple shader, which just takes the rgb from src0, and the alpha from src1 (breaks the same way, only one texture is used for both inputs). But as adding the two rgba sources does work correctly, this has to be a bug in the shader parser and/or compiler?

Steps to Reproduce

You may be able to repro simply, but my steps were:

  1. Install obs-multisource-effect (https://obsproject.com/forum/resources/multi-source-effect.1412/)
  2. duplicate its add.effect to [custom name].effect
  3. make the indicated changes to output only src0 or src1 (while still sampling both)
  4. apply the plugin, set two video or image sources and use the custom shader file.

result: outputting src1 results in src0 instead (as detailed).

Anything else we should know?

again note that in the example, if the sampler line for src0 is removed, then src1 is output correctly.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the obs-multisource-effect add.effect example and reproduce the minimal shaders that sample src0 and src1. Then trace the OBS shader parser/compiler path involved in compiling those samples. Done means returning src1 still produces src1 when src0 is also sampled, while the existing combined-texture behavior remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
compilers, computer-graphics
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.