OpenGL shader bug?

Start by reproducing the behavior with mpv git master on the stated Raspberry Pi and compare the two fragment shaders with the EGL dispmanx alpha configuration. Check whether the visible console and letterboxed clear regions correlate with fragment alpha handling when GL_BLEND is disabled. Done means isolating the source of the unexpected compositing and confirming the result across the shader and video-overlay paths.

I'm seeing very weird behavior with RPI apparently alpha-blending the EGL framebuffer with the dispmanx layers behind it (on Raspbian that's the Linux console with the shell).

Working shader, which renders a solid, opaque deep shade of green:

#version 100
precision mediump float;
precision mediump sampler2D;
#define texture texture2D
varying vec2 texcoord0;
varying vec2 texcoord1;
varying vec2 texcoord2;
varying vec2 texcoord3;
varying vec2 texcoord4;
varying vec2 texcoord5;
uniform mat3 colormatrix;
uniform vec3 colormatrix_c;
uniform sampler2D texture0;
uniform vec2 texture_size0;
uniform mat2 texture_rot0;
uniform vec2 pixel_size0;
uniform sampler2D texture1;
uniform vec2 texture_size1;
uniform mat2 texture_rot1;
uniform vec2 pixel_size1;
uniform sampler2D texture2;
uniform vec2 texture_size2;
uniform mat2 texture_rot2;
uniform vec2 pixel_size2;
#define LUT_POS(x, lut_size) mix(0.5 / (lut_size), 1.0 - 0.5 / (lut_size), (x))
void main() {
vec4 color = vec4(0.0, 0.0, 0.0, 1.0);
color.r = 1.000000 * vec4(texture(texture0, texcoord0)).r;
color.g = 1.000000 * vec4(texture(texture1, texcoord1)).r;
color.b = 1.000000 * vec4(texture(texture2, texcoord2)).r;
// color conversion
color.rgb = mat3(colormatrix) * color.rgb + colormatrix_c;
vec4 tmp = color;
color = vec4(tmp.r * 0.0, 0.2, 0, 1.0);
// color mapping
gl_FragColor = color;
}

Not working, the console is very visible on top of the "video":

#version 100
precision mediump float;
precision mediump sampler2D;
#define texture texture2D
varying vec2 texcoord0;
varying vec2 texcoord1;
varying vec2 texcoord2;
varying vec2 texcoord3;
varying vec2 texcoord4;
varying vec2 texcoord5;
uniform mat3 colormatrix;
uniform vec3 colormatrix_c;
uniform sampler2D texture0;
uniform vec2 texture_size0;
uniform mat2 texture_rot0;
uniform vec2 pixel_size0;
uniform sampler2D texture1;
uniform vec2 texture_size1;
uniform mat2 texture_rot1;
uniform vec2 pixel_size1;
uniform sampler2D texture2;
uniform vec2 texture_size2;
uniform mat2 texture_rot2;
uniform vec2 pixel_size2;
#define LUT_POS(x, lut_size) mix(0.5 / (lut_size), 1.0 - 0.5 / (lut_size), (x))
void main() {
vec4 color = vec4(0.0, 0.0, 0.0, 1.0);
color.r = 1.000000 * vec4(texture(texture0, texcoord0)).r;
color.g = 1.000000 * vec4(texture(texture1, texcoord1)).r;
color.b = 1.000000 * vec4(texture(texture2, texcoord2)).r;
// color conversion
color.rgb = mat3(colormatrix) * color.rgb + colormatrix_c;
vec4 tmp = color;
color = vec4(tmp.r * 0.01, 0.2, 0, 1.0);
// color mapping
gl_FragColor = color;
}

Sorry for the redundant uniforms. As you can see, in both cases gl_FragColor.a will be 1.0, i.e. opaque. The EGL dispmanx layer was created with

    VC_DISPMANX_ALPHA_T alpha = {
        .flags = DISPMANX_FLAGS_ALPHA_FROM_SOURCE,
        .opacity = 0xFF,
    };

and GL_BLEND is disabled.

This issue is especially visible if the video is letterboxed - then the regions cleared by glClear() are opaque, while the regions drawn over by the video are not.

This can be seen with mpv (git master). Normally it does color.a = 1.0;. mpv has pretty complex shader generation; this patch touches the affected code: http://sprunge.us/ghUd (if you change 0.01 to 0.0 the background console will disappear).

In both shader dumps I pasted above, the normal video shader was slightly edited for test purposes. In normal operation, the problem becomes visible if the mmal video overlay is disabled and shaders are used for video - then the video appears faded and gray, with the background console being visible.

I suspect OSD rendering might have similar problems, but haven't confirmed.

Version infos:

Linux raspberrypi 4.4.22-v7+ #912 SMP Mon Sep 26 19:04:59 BST 2016 armv7l GNU/Linux

Sep 21 2016 13:13:29 
Copyright (c) 2012 Broadcom
version 2eaf52cc53435b5ce67253af1487f9a4f9f96e2d (clean) (release)

All on Raspbian... I ran rpi-update before making a final test.