Stereo 3D layout signalled in the stream (H.264 frame packing SEI, MP4 st3d) is ignored

mpv Information

mpv v0.41.0, libplacebo v7.360.1 — Debian 14, x86_64, libavcodec 63.1.101.

Summary

mpv detects a stereo 3D layout only from the Matroska StereoMode element. A layout
signalled in the video stream — the H.264/HEVC frame_packing_arrangement SEI (payload type
45), or the MP4 st3d box — is ignored, and such files play as stereo-in=mono.

libavcodec already decodes that SEI into AVStereo3D frame side data and hands it to mpv. mpv
never asks for it.

Reproduction Steps

Three files isolating each signal, built in three commands:

ffmpeg -f lavfi -i "testsrc2=size=960x1080:rate=24:duration=2" \
       -f lavfi -i "smptebars=size=960x1080:rate=24:duration=2" \
       -filter_complex "[0:v][1:v]hstack=inputs=2[v]" -map "[v]" \
       -c:v libx264 -crf 20 -pix_fmt yuv420p source_sbs.mp4

ffmpeg -i source_sbs.mp4 -c:v copy -an plain.mkv                      # no signal
mkvmerge -o container_only.mkv --stereo-mode 0:1 plain.mkv            # container tag only
ffmpeg -i source_sbs.mp4 -c:v libx264 -crf 20 -pix_fmt yuv420p \
       -x264-params frame-packing=3 sei_only.mp4                      # in-band SEI only

for f in plain.mkv container_only.mkv sei_only.mp4; do
  printf '%-20s ' "$f"
  mpv --no-config --vo=null --frames=1 --no-audio \
      --term-playing-msg='${video-params}' "$f" 2>/dev/null | grep -i stereo-in
done

Actual Behavior

plain.mkv            stereo-in=mono
container_only.mkv   stereo-in=sbs2l
sei_only.mp4         stereo-in=mono     <-- expected sbs2l

Expected Behavior

sei_only.mp4 should report stereo-in=sbs2l. The two files are the same picture carrying the
same layout declared two different ways.

The data is present and correct

The SEI is in the stream — trace_headers parses it:

$ ffmpeg -v trace -i sei_only.mp4 -map 0:v:0 -frames:v 2 -c copy \
      -bsf:v trace_headers -f null - 2>&1 | grep frame_packing
frame_packing_arrangement_id                 1 = 0
frame_packing_arrangement_cancel_flag        0 = 0
frame_packing_arrangement_type         0000011 = 3      # 3 = side by side
frame_packing_arrangement_repetition_period  010 = 1

And libavcodec already converts it to AVStereo3D and attaches it to the decoded frame:

$ ffmpeg -v info -i sei_only.mp4 -vf showinfo -frames:v 1 -f null -
[Parsed_showinfo_0] side data - Stereo 3D: type - side by side, view - packed, primary_eye - none

So by the time mpv receives the AVFrame, the layout is already decoded, normalised and
attached.

Diagnosis

stereo3d has exactly one source in the tree. demux/demux_mkv.c:752 reads the Matroska
element, and no other demuxer sets it — grep -rn 'stereo3d\|StereoMode\|stereo_mode' demux/*.c
matches only demux_mkv.c.

mp_image_from_av_frame() (video/mp_image.c) imports nine kinds of frame side data from
libavcodec — AV_FRAME_DATA_DISPLAYMATRIX, ICC_PROFILE, MASTERING_DISPLAY_METADATA,
CONTENT_LIGHT_LEVEL, DYNAMIC_HDR_PLUS, A53_CC, DOVI_METADATA, DOVI_RPU_BUFFER,
FILM_GRAIN_PARAMS — but not AV_FRAME_DATA_STEREO3D.

Additionally, fix_image_params() in filters/f_decoder_wrapper.c overwrites the layout on
every frame with the container tag from the demuxer.

Why reading the side data is not enough on its own

Worth recording, because it is the non-obvious half of the bug. The in-band signal is attached
only to the frames that carry it — measured on the files above:

file	frames with Stereo 3D side data	keyframes
sei_only.mp4 (48 frames)	1	1
a real SBS MP4 (first 200 frames)	4	4

An exact 1:1 match with keyframes, consistent with frame_packing_arrangement_repetition_period = 1 ("persists until the next IDR"). So reading the side data alone still produced mono: the
keyframe was detected, the following frame carried no side data, fix_image_params() ran again
and reset the format. The layout has to persist across the frames that do not repeat it.

Relation to existing issues

• #1045 ("auto-detecting 3D input format") is closed and predates AVStereo3D being wired
  through this path.
• #18348 concerns the removed full-width sbsl/abl choices; its reporter's file is a
  Matroska whose tag mpv reads correctly. That is the layout vocabulary; this is the
  detection source.

Why this is becoming more common

HandBrake merged support for writing this SEI
(PR #8100), and FFmpeg has an open request
for a lossless injector for it
(#24531), so files carrying the in-band
signal are going to keep appearing.

Fix

A tested two-part patch is ready and I will open a pull request referencing this issue: read
AV_FRAME_DATA_STEREO3D in mp_image_from_av_frame(), and remember an in-band layout across
the frames that do not repeat it in f_decoder_wrapper.c, falling back to the container tag
only when the stream signals nothing.

Verified on a patched build: sei_only.mp4 goes from mono to sbs2l, three real SEI-only
MP4s go from mono to sbs2l/sbs2r, and no container-tagged or unsignalled file changes
behaviour. One of those files declares side by side (inverted) — right eye first — while its
filename says SBS, so it is also a case where guessing from the filename would swap the
viewer's eyes.

---

Disclosure

An AI partner helped produce the outcomes and this post.

• Harness: Claude Code (Anthropic's agentic CLI)
• Model class: Claude Opus (Anthropic frontier tier)
• Model: Claude Opus 5 (1M context), model ID claude-opus-5[1m]

Every result above was executed and measured on the machine described, not inferred: the patch
was compiled, the before/after values come from running both binaries against the same files,
and the side-data-per-keyframe counts were measured with ffmpeg -vf showinfo.

Start with mp_image_from_av_frame() in video/mp_image.c and fix_image_params() in filters/f_decoder_wrapper.c, then run the reproduction commands against the reported files. Done means the in-band SEI case reports the correct stereo layout persistently, while container-tagged and unsignalled files retain their existing behavior.