mltframework / mltframework/mlt

Color shift after first frame of output when using two composite transitions

Open
#623 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

confirmed workaround
Dominant language
C
Stars
1.9k
Forks
387
Avg merge
17h 28m
Merged PRs (30d)
8

Description

Hi there! I'm diving into MLT to composite two videos next to each other at specific sizes and orientations. They may animate between different orientations, but for this issue, I kept it to just one to demonstrate the problem. The problem is that I often see a noticeable color shift right after the video starts. If there's no transition, I don't see a color shift in the output. (The color is slightly different from the original, but maybe close enough to get by. But what I mean is, it doesn't noticeably change from the first frame to the second.)

I can get rid of the shift if I switch the profile colorspace from 709 to 601, though I'm not sure if that means all but the first frame is output in 601 or if the smaller colorspace is just masking the color shift. The output does seem slightly darker than the original, at least in this example.

I was initially having this issue using the mlt library via swig, but I was also able to reproduce it with melt. The input XML in my example was created using the XML consumer from a ruby script, hence quite a bit of unnecessary attributes. But I included it all for completeness.

Ideally my output would be in the 709 colorspace and be as true to the original colors as possible while also having no discernible color shift in the output. Am I doing that correctly or is there a better way? What am I missing?

The input file: https://embed-ssl.wistia.com/deliveries/d2e5245e04b6bcdce7981a5bb550b602/pets000.mov?disposition=attachment

The output file (you can see the color shift here): https://embed-ssl.wistia.com/deliveries/0fa094f2a99c63b1253f01d7675082ef/output.mov?disposition=attachment

The command I'm using:

melt \
  xml:input.xml \
  -consumer \
    avformat:output.mov \
    vcodec=libx264 \
    f=mov \
    crf=23 \
    x264opts=keyint=89:min-keyint=89:scenecut=-1 \
    movflags=faststart \
    strict=strict \
    threads=0 \
    width=1280 \
    height=720

input.xml:

<?xml version="1.0"?>
<mlt LC_NUMERIC="en_US.UTF-8" version="6.23.0">
  <profile description="DV/DVD PAL" width="1280" height="720" progressive="1" sample_aspect_num="1" sample_aspect_den="1" display_aspect_num="1" display_aspect_den="1" frame_rate_num="30" frame_rate_den="1" colorspace="709"/>
  <producer id="producer0" in="0" out="114">
    <property name="length">15000</property>
    <property name="eof">continue</property>
    <property name="resource">0x000000ff</property>
    <property name="aspect_ratio">1</property>
    <property name="mlt_service">colour</property>
    <property name="colour">0x000000FF</property>
  </producer>
  <producer id="producer1" in="0" out="114">
    <property name="length">358</property>
    <property name="eof">continue</property>
    <property name="resource">pets000.mov</property>
    <property name="meta.media.nb_streams">2</property>
    <property name="meta.media.0.stream.type">video</property>
    <property name="meta.media.0.stream.frame_rate">23.976</property>
    <property name="meta.media.0.stream.sample_aspect_ratio">0</property>
    <property name="meta.media.0.codec.width">1280</property>
    <property name="meta.media.0.codec.height">720</property>
    <property name="meta.media.0.codec.rotate">0</property>
    <property name="meta.media.0.codec.frame_rate">24000</property>
    <property name="meta.media.0.codec.pix_fmt">yuv420p</property>
    <property name="meta.media.0.codec.sample_aspect_ratio">0</property>
    <property name="meta.media.0.codec.colorspace">709</property>
    <property name="meta.media.0.codec.color_trc">1</property>
    <property name="meta.media.0.codec.name">h264</property>
    <property name="meta.media.0.codec.long_name">H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10</property>
    <property name="meta.media.0.codec.bit_rate">5948147</property>
    <property name="meta.attr.0.stream.language.markup">eng</property>
    <property name="meta.attr.0.stream.handler_name.markup">Apple Video Media Handler</property>
    <property name="meta.media.1.stream.type">audio</property>
    <property name="meta.media.1.codec.sample_fmt">s16</property>
    <property name="meta.media.1.codec.sample_rate">44100</property>
    <property name="meta.media.1.codec.channels">2</property>
    <property name="meta.media.1.codec.name">pcm_s16le</property>
    <property name="meta.media.1.codec.long_name">PCM signed 16-bit little-endian</property>
    <property name="meta.media.1.codec.bit_rate">1411200</property>
    <property name="meta.attr.1.stream.language.markup">eng</property>
    <property name="meta.attr.1.stream.handler_name.markup">Apple Sound Media Handler</property>
    <property name="meta.attr.major_brand.markup">qt  </property>
    <property name="meta.attr.minor_version.markup">512</property>
    <property name="meta.attr.compatible_brands.markup">qt  </property>
    <property name="meta.attr.encoder.markup">Lavf58.20.100</property>
    <property name="seekable">1</property>
    <property name="meta.media.sample_aspect_num">1</property>
    <property name="meta.media.sample_aspect_den">1</property>
    <property name="aspect_ratio">1</property>
    <property name="audio_index">1</property>
    <property name="video_index">0</property>
    <property name="mute_on_pause">1</property>
    <property name="mlt_service">avformat</property>
  </producer>
  <producer id="producer2" in="0" out="114">
    <property name="length">358</property>
    <property name="eof">continue</property>
    <property name="resource">pets000.mov</property>
    <property name="meta.media.nb_streams">2</property>
    <property name="meta.media.0.stream.type">video</property>
    <property name="meta.media.0.stream.frame_rate">23.976</property>
    <property name="meta.media.0.stream.sample_aspect_ratio">0</property>
    <property name="meta.media.0.codec.width">1280</property>
    <property name="meta.media.0.codec.height">720</property>
    <property name="meta.media.0.codec.rotate">0</property>
    <property name="meta.media.0.codec.frame_rate">24000</property>
    <property name="meta.media.0.codec.pix_fmt">yuv420p</property>
    <property name="meta.media.0.codec.sample_aspect_ratio">0</property>
    <property name="meta.media.0.codec.colorspace">709</property>
    <property name="meta.media.0.codec.color_trc">1</property>
    <property name="meta.media.0.codec.name">h264</property>
    <property name="meta.media.0.codec.long_name">H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10</property>
    <property name="meta.media.0.codec.bit_rate">5948147</property>
    <property name="meta.attr.0.stream.language.markup">eng</property>
    <property name="meta.attr.0.stream.handler_name.markup">Apple Video Media Handler</property>
    <property name="meta.media.1.stream.type">audio</property>
    <property name="meta.media.1.codec.sample_fmt">s16</property>
    <property name="meta.media.1.codec.sample_rate">44100</property>
    <property name="meta.media.1.codec.channels">2</property>
    <property name="meta.media.1.codec.name">pcm_s16le</property>
    <property name="meta.media.1.codec.long_name">PCM signed 16-bit little-endian</property>
    <property name="meta.media.1.codec.bit_rate">1411200</property>
    <property name="meta.attr.1.stream.language.markup">eng</property>
    <property name="meta.attr.1.stream.handler_name.markup">Apple Sound Media Handler</property>
    <property name="meta.attr.major_brand.markup">qt  </property>
    <property name="meta.attr.minor_version.markup">512</property>
    <property name="meta.attr.compatible_brands.markup">qt  </property>
    <property name="meta.attr.encoder.markup">Lavf58.20.100</property>
    <property name="seekable">1</property>
    <property name="meta.media.sample_aspect_num">1</property>
    <property name="meta.media.sample_aspect_den">1</property>
    <property name="aspect_ratio">1</property>
    <property name="audio_index">1</property>
    <property name="video_index">0</property>
    <property name="mute_on_pause">1</property>
    <property name="mlt_service">avformat</property>
  </producer>
  <tractor id="tractor0" global_feed="1" in="" out="">
    <track producer="producer0"/>
    <track producer="producer1"/>
    <track producer="producer2"/>
    <transition id="transition0" out="114">
      <property name="a_track">0</property>
      <property name="b_track">1</property>
      <property name="start">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
      <property name="factory">loader</property>
      <property name="aligned">1</property>
      <property name="progressive">1</property>
      <property name="mlt_service">composite</property>
      <property name="end">-213.33333333333337/120.0:853.3333333333335x480.00000000000006</property>
      <property name="distort">1</property>
    </transition>
    <transition id="transition1" out="114">
      <property name="a_track">0</property>
      <property name="b_track">2</property>
      <property name="start">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
      <property name="factory">loader</property>
      <property name="aligned">1</property>
      <property name="progressive">1</property>
      <property name="mlt_service">composite</property>
      <property name="end">426.66666666666663/120.0:853.3333333333335x480.00000000000006</property>
      <property name="distort">1</property>
    </transition>
  </tractor>
</mlt>

I compiled mlt from commit 0e756b692b4b1248d27254d124cd41f580878372--I upgraded from 16.6.0 where I had the same issue--with this configuration:

./configure --disable-decklink --disable-dv --disable-feeds --disable-frei0r --disable-gtk2 --disable-jackrack --disable-kdevlive --disable-kino --disable-linsys --disable-lumas --disable-motion_est --disable-ndi --disable-normalize --disable-oldfilm --disable-opencv --disable-opengl --disable-qt --disable-resample --disable-rtaudio --disable-sox --disable-swfdec --disable-vid.stab --disable-videostab --disable-vmfx --disable-vorbis --disable-xine --swig-languages=ruby

And I'm on Mac OS Catalina 10.15.7.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the report with input.xml and the supplied melt command, comparing the first frames with and without the two composite transitions and with profile colorspace 709. Trace the color-processing path used by those transitions; done means output in 709 no longer has a visible shift after the first frame while preserving the reported composite layout.

Written by the indexing model from the issue text.

Assessment

Tech stack
c
Domain
audio-video-rtc
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.