HLS: Support TTML/IMSC subtitles (in mp4 segments)
@icbaker is already working on this.
Since Aug 18, 2023.
- Dominant language
- Java
- Stars
- 3k
- Forks
- 955
- Avg merge
- 12d 14h
- Merged PRs (30d)
- 2
Description
I'm trying to figure out a way to make time-based subtitles captured from a live DASH stream display correctly after converting the stream to HLS. I'm using ExoPlayer v2.18.6.
My app converts a live DASH stream into HLS. A/V works as expected, but the DASH stream includes TTML-formatted subtitle segments that do not display. The app stores A/V segments from the live DASH stream, beginning at a random time within the stream. Audio, video, and subtitle segments are stored. I then create a set of HLS playlists from these segments to play back the video.
The TTML subtitle segments contain "begin" and "end" times for the display text relative to the start of the live DASH stream. A typical subtitle segment includes something like:
<p begin="489:02:17.897" end="489:24:18.698" region="Region_187">
<span style="Style_184_0">WHAT SHOULD I DO?</span>
</p>
When playing the recording as a DASH stream, the subtitles display correctly, even when played back as a static DASH stream; i.e. not a dynamic stream using clock timing relative to when the stream was originally played. This means the subtitle decoder must be getting the timing data from the A/V stream and displaying the subtitles at the correct time based on this. (Side note: I see other evidence that this is true, like ExoPlayer's timeline bar has 489:20:17 as a start time.)
When playing the segments as an HLS stream, however, this internal timing reference is apparently NOT used, and the subtitles do not display, because the playback start time is 000:00:00 and the first subtitle displays at some much later time (489 hours, in the above example).
If I manually adjust the times in the segments, I get mixed results on whether or not they display. For example, if the above subtitle is the first one at the beginning of the stream, I can adjust it to look like this:
<p begin="000:00:00.000" end="000:00:00.801" region="Region_187">
<span style="Style_184_0">WHAT SHOULD I DO?</span>
</p>
In some cases the caption will display correctly, and others it may just blink on and back off again. My hunch is that I'm not adjusting the times correctly according to when they should be displayed based on the A/V timing. Assuming that the first subtitle should start at time 0 rarely works as desired.
I'm wondering if there is a way to extract the display time value from the MP4 data and use that as a reference start time, and then adjust the subtitle times by this value. I'm not sure what this value would be called or where to look for it; I don't know much about how the the A/V renderers operate. I considered using the DASH timing information (availability start time, current time, etc), but I don't think this will be accurate enough.
Here is an example of the and HLS stream (as extracted from a live DASH stream) with TTML subtitles:
https://1drv.ms/u/s!Ar-5IRkcwwk9gddDSA3TSrBPLus2Sw?e=QhVpC4
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.