Live DASH: Audio starts 1-5s before video due to manifest timeline vs. media PTS misalignment

1. TL;DR

We are experiencing an issue with a live DASH streams where audio playback begins 1-5 seconds before any video frames are rendered, resulting in an initial black screen for the user.

This appears to be primarily caused by a significant and drifting misalignment between the manifest's segment timeline (<S t=... > values) and the actual presentation timestamps (PTS) within the media segments.

This problem is further complicated by the absence of a presentationTimeOffset in the manifest, and the fact that audio and video segments have slightly different durations and are not time-aligned.

We would appreciate your expert opinion on whether ExoPlayer can be configured or has internal mechanisms to mitigate this behavior, or if this type of misalignment falls outside the player's scope and must be addressed at the stream generation level.

2. Problem Description & Root Causes

The core problem manifests as a period of up to 5 seconds (1 segment) at the start of playback where audio is audible, but the video screen remains black.

The identified root causes are:

• Manifest Timeline vs. Media PTS Misalignment: The timestamps (t attributes in <S> elements within SegmentTimeline) in the DASH manifest are significantly offset from the presentation timestamps embedded within the media segments. This offset can be very large (e.g., 30+ hours) and is not consistent.
  • No presentationTimeOffset: The manifest does not provide a presentationTimeOffset in the Period element, which would typically help align the MPD timeline with the media timeline.
  • PTS Drift: Over extended periods (streams can run for weeks), the presentation timestamps in the media segments drifts relative to the real-time clock and potentially relative to the manifest's own timeline progression if it were ideally aligned. This drift accumulates, exacerbating the initial misalignment.
  • We know that this isn't allowed in the Dash spec (max is 50% of a segment) but avoiding such drift is very difficult and we would like to explore a Player solution
• Unaligned Audio/Video Segments: Audio and video segments have slightly different durations (~10ms) and their start times are not aligned.
  Exaggerated example of the audio vs video segment timeline:

  |-----|-----|-----|-----|-----|-----|-----| video
  |------|------|------|------|------|------| audio
  

3. Analysis & Observations

Our understanding of ExoPlayer's behavior in this scenario is as follows:

• a. ExoPlayer downloads the manifest.
• b. It picks a "safe" offset from the live edge (based on the manifest timeline) to start playback, aiming to be neither too far nor too close to the live window edge.
• c. It downloads the video and audio segments that correspond to the position calculated in step (b) according to the manifest timeline.
• d. Because the presentation timestamps of the downloaded segments (from step c) do not match the target start position derived from the manifest timeline (from step b), there's a discrepancy.
• e. It appears ExoPlayer then defaults to starting playback from the beginning of the available buffered audio data. (We are unsure of the exact internal logic here).

Example (segment |---| is only in the timeline, |___| is initial download):

|-----|-----|-----|_____|-----|-----|-----| video
|------|------|______|------|------|------| audio
                  ^ initial position (for download) (with a 30h offset)
              ^ playback starts there (only audio)
                     ^ video starts here

Since there are no video frames with presentation timestamps matching the very beginning of the buffered audio data (due to the audio & video different duration), the user sees a black screen until the playback position reaches a timestamp for which a video frame is available in the buffered video data. This delay can be up to the duration of a full video segment.

Manifest Snippets:

Note the large t values in SegmentTimeline compared to segment startNumber

<SegmentTemplate timescale="90000" media="audio/audio-$Number$.mp4" startNumber="4858">
 <SegmentTimeline>
  <S t="2191815264" d="451200"/>
  <S t="2192266464" d="451200"/>
  </SegmentTimeline>
</SegmentTemplate>
...
<SegmentTemplate timescale="90000" media="video/1/video-$Number$.mp4" startNumber="4866" >
  <SegmentTimeline>
    <S t="2191884681" d="450450"/>
    <S t="2192335131" d="450450"/>
  </SegmentTimeline>
</SegmentTemplate>

(Full mpdFileName.mpd.)

Media Segment Timestamps (from mp4dump):

Observe the earliest_presentation_time and base_media_decode_time values, which are vastly different from the manifest t values. For example, audio-4858.mp4 has a manifest t="2191815264" (approx 6.7 hours if timescale is 90000 and assuming t is in timescale units) but an actual earliest_presentation_time="9833019849" (approx 30.3 hours).

$ mp4dump audio/audio-4858.mp4
[sidx]
  timescale = 90000
  earliest_presentation_time = 9833019849
[moof/traf/tfdt]
  base media decode time = 9833019849

$ mp4dump video/1/video-4866.mp4
[sidx]
  timescale = 90000
  earliest_presentation_time = 9833089266
[moof/traf/tfdt]
   base media decode time = 9833089266

Also notice the difference between audio-4858.mp4 earliest_presentation_time (9833019849) and video-4866.mp4 earliest_presentation_time (9833089266) is 70017 (or ~0.77 seconds at 90k timescale).
In practice it continuously varies as audio segments are ~10ms longer.

4. Attempted Workarounds

We attempted to mitigate this by forcing ExoPlayer to download the previous video segment relative to its initial choice. We tested this by patching the dash manifest on the fly to decrement startNumber of the video SegmentTemplate by one.

This resulted in a similar issue, with playback starting at the beginning of the audio buffer and the video queue playing in fast forward until the current position (the "fast forward" is due to shouldForceRenderOutputBuffer 100ms force render).

|-----|-----|_____|-----|-----|-----|-----| video (shifted segment timeline by 1 segment)
|------|------|______|------|------|------| audio
                  ^ initial position chosen by player (based on manifest)
              ^ playback starts here (audio only, due to media timestamps)
            ^ Video starts here and play fast to catch-up playback position

5. Reproduction & Stream Availability

Unfortunately, we cannot provide a publicly accessible stream URL for reproduction. The streams are generated by specialized hardware (a set-top box receiving a satellite feed and transforming it into DASH) that requires a specific physical setup (midleware box & customer STB) and is not available externally.

We are happy to provide any additional logs, data, or perform specific tests on request.

6. Questions for the ExoPlayer Team

1. Given the significant and drifting offset between manifest timeline and media PTS, and the absence of presentationTimeOffset, how does ExoPlayer determine the initial synchronization point for playback? Is our observation that it defaults to the start of the buffered audio accurate in such scenarios?
2. Are there any existing ExoPlayer configurations that could be used to force the start position to be the max of the first video & first audio frames?
3. Could a custom DashChunkSource or modifications to how DashMediaPeriod processes segment information potentially allow us to inject corrected timing information or influence the initial segment selection based on actual media PTS rather than solely manifest timeline?
4. Is this type of stream fundamentally "broken" from a player's perspective, meaning the inconsistencies are too severe to be reliably handled client-side and must be fixed at the encoder/packager level?

Thank you for your time and assistance.