Adaptive Bitrate Streaming Proposal/Specification

Based on #4720

This issue drafts a high-level proposal to add support for Adaptive Bitrate Streaming in the Joystream network.

## Definitions
- **_ABR:_** Adaptive bitrate streaming.
- **_Transcoding:_** conversion of one type of encoded data (video or audio) to another. This can involve changing the file type, the codec, or the resolution.
- **_Packaging:_** Process of generating the DASH/HLS manifest files by providing multiple audio & video streams as input.
- **_Bitrate:_** The number of bits that are conveyed or processed per sec by the video player. In the context of streaming, for smooth playback of VOD(video-on-demand) streams, the bandwidth of the client network should be >= to the bitrate of the stream. Otherwise, the playback would buffer.
- **_GOP:_** Group of Pictures ([GOP](https://aws.amazon.com/blogs/media/part-1-back-to-basics-gops-explained/)) is a concept used in video compression, specifically in the context of codecs. GOP is a sequence of consecutive video frames that are grouped together for the purpose of more efficient video encoding and decoding. While encoding, the encoders choose two frames (called **_I-frames_** or Intra-coded frame) which are self-contained & don't depend on any other frame while decoding, all the frames between I-frames are not self-contained and are predicted/reconstructed by the decoder based on the differences between the previous frames. The I-frame & all the frames between it forms the GOP - the smallest unit of a video segment that can be independently played. For example, if GOP is 90 frames in a video with 30 frames/sec, then the GOP length is 3 sec (smallest independent playable unit).
- **_Native resolution:_** The native resolution of a screen refers to the physical number of vertical and horizontal pixels contained within it. If the video resolution is more than the native resolution of the device display, then the videos are scaled down a bit to fit the display resolution of the device. So in the context of streaming, even if the network could handle the high bitrate stream, the playback experience could be worse given the continuous processing cost associated with downscaling.

# DASH vs HLS

Both protocols are widely supported and used. HLS was developed first by Apple for its devices, while, the dash is a newer open-source protocol and is becoming compatible with many platforms eventually.

- Both DASH and HLS are http-based adaptive streaming protocols
- Segmented Delivery: Both DASH and HLS deliver content in small segments, typically of a few seconds each. This allows them to quickly adapt to network conditions or device capabilities changes.
- Multi-Codec Support: Both DASH and HLS support multiple codecs, allowing the content providers to choose the one that best suits their needs.
- Fallback Quality Levels: Both DASH and HLS allow for different quality levels to be prepared and served depending on the end user's bandwidth and device capabilities.
- Redundancy: Both DASH and HLS support multiple redundant streams for high availability and fault tolerance. If one stream fails, the player can automatically switch to another.

Here is a table comparing features b/w DASH & HLS:
| Feature | HLS | MPEG-DASH |
|-----------------------------------|---------------------------------------------|----------------------------------------------|
| Created by | Apple Inc. (proprietary protocol) | A consortium of companies led by MPEG |
| Native HTML5 Playback | Only in Safari and Edge | Yes via MSE(Media Source Extensions) API |
| Supported by Safari browser | Yes | No (Still possible to play using client-side libraries e.g. `dash.js`) |
| Niatve iOS support | Yes (only HLS is supported) | No |
Compatible Codecs | Video: H.264, H.265  Audio: AAC-LC, HE-AAC+ v1 & v2, xHE-AAC, Apple Lossless, FLAC | Codec agnostic for both audio and video
Container Formats | Has traditionally used MPEG-2 or MPEG-TS | Has traditionally used MP4/.mp4
Manifest file format | Text file with `. m3u8` extension | an XML file with `. mpd` extension

In short, apart from differences in device/codec compatibility support, both protocols are being widely used and many video platforms (e.g. Youtube) support both. In fact, few packaging software provides the option of generating both DASH & HLS manifests using the same list of video resolution streams as input.

So for the purpose of this writeup, only DASH is being discussed but both have simpler features, and understanding one in detail would provide sufficient information on working of ABR streaming.

## Structure of the DASH manifest (`.mpd` file)
- The DASH Media Presentation contains a sequence of one or more _Periods_.
- A _Period_ contains one or more _Adaptation Sets_.
- An _Adaptation Set_ contains one or more _Representations_.
- A _Representation_ contains one or more _Segments_.
- _Segments_ carry the actual media data and associated metadata.

Here is an excellent [article](https://ottverse.com/structure-of-an-mpeg-dash-mpd/) describing the structure of the DASH manifest in detail. Also an [image](https://www.cablelabs.com/wp-content/uploads/2015/05/DASH-MPD-723x1024.png) to show the structure of DASH manifest files

Following is a hierarchical description of different DASH manifest tags & attributes:

- **MPD** (Media Presentation Description)
- _minBufferTime_: The minimum amount of time that a client must buffer before starting playback. This is important for ensuring that playback is smooth and uninterrupted, even if there are brief interruptions in the network connection.
- _type_: The type of the MPD (e.g., "static" for VOD, "dynamic" for live streaming).
- _mediaPresentationDuration_: The total duration of the media presentation.
- _profiles_: The profile of the MPD which indicates the features of DASH the MPD adheres to.
- **BaseURL**: The URL that serves as the base for resolving relative URLs within the manifest. Multiple URLs can be provided to ensure reliability if one URL is not working
- **Period**
- _id_: A unique identifier for the period.
- _duration_: The duration of the period.
- **AdaptationSet**
- _id_: A unique identifier for the adaptation set.
- _maxWidth_: The maximum width (in pixels) of the video frames in the adaptation set.
- _maxHeight_: The maximum height (in pixels) of the video frames in the adaptation set.
- _par_: The pixel aspect ratio of the video in the adaptation set.
- _lang_: The language of the content in the adaptation set.
- **Representation**
- _id_: A unique identifier for the representation.
- _mimeType_: The MIME type of the content in the representation.
- _codecs_: The codec used to encode the content in the representation.
- _width_: The width (in pixels) of the video frames in the representation.
- _height_: The height (in pixels) of the video frames in the representation.
- _frameRate_: The frame rate of the video in the representation.
- _sar_: The sample aspect ratio of the video in the representation.
- _bandwidth_: The average bandwidth (in bits per second) required for the representation.
- **SegmentBase**
- _indexRange_: Specifies the byte range within the media file for the segment index.
- **Initialization**
- _range_: Specifies the byte range within the media file for the initialization segment.

Examples of a single period, two adaption sets (one audio, one video), and two representations (resolutions) `.mpd` file

```xml

https://example-1.com/assets
https://example-2.com/assets



320x180.mp4





640x360.mp4








audio.mp4

```

# Runtime Changes ???

- I think no runtime change is required if we decide to use a collaborator member approach to register the media assets on-chain after transcoding. The channel owner could provide `ManageVideoAssets` permission if they want to opt-in for transcoding their assets and made them available for ABR. This permission is more restrictive than `AddVideo`, and only allows updating the video assets using `update_video` extrinsic. Obviously, this assumes that the entry app would be paying all the transaction costs associated with registering assets on-chain. Using this approach this video can be instantly created on a chain with the original assets. And then Orion can do the registering/uploading of transcoded assets as they become available, which could take a while depending upon the backlog of all assets across the different channels that need to be transcoded.
- Alternative approach which does not require collaborative permission, is based on user account-based transactions & notifications. First, the client would submit assets uploaded by the creator to Orion's uploads endpoint (say `/uploads`). Next, Orion would perform media transcoding, and when it's ready it would notify the client(that the processing has been done) along with the encoded transaction(with all transcoded assets) in the notification response. Now, the creator can publish the Video.

# Metadata Change

## Current Schema
Currently, there is a one-to-one relationship between video and it's media data object.

```gql
type Video @entity {
"Runtime identifier"
id: ID!

# ...

"Video media asset"
media: StorageDataObject

"Video file metadata"
mediaMetadata: VideoMediaMetadata
}

type StorageDataObject @entity {
"Data object runtime id"
id: ID!

# ...

"Video that has this data object associated as media."
videoMedia: Video @derivedFrom(field: "media")
}
```

## Proposed Schema
- To add the support for multiple resolution media assets in video entity, the video <-> dataObject relation should be changed from one-to-one to one-to-many.
Also, in the current approach, the `mediaMetadata` field entity is directly associated with the `Video` entity, but in the proposed scheme the metadata information is part of the `StorageDataObject` entity so it's possible to identify the metadata with each media file. e.g., media type (audio/video), resolution, codec, etc.

```gql
type Video @entity {
"Runtime identifier"
id: ID!

# ...

"Video media assets"
mediaStreams: [StorageDataObject!]! @derivedFrom(field: "videoMedia")
}

type StorageDataObject @entity {
"Data object runtime id"
id: ID!

# ...

"Video that has this data object associated as media."
videoMedia: Video @derivedFrom(field: "media")

"Video file metadata"
mediaMetadata: VideoMediaMetadata
}
}
```

# Orion changes

## Video transcoding
Orion needs to support the media transcoding feature, which should do the work in a scalable way (proposal WIP)

## Assets upload
Orion needs to support the uploading of transcoded assets to storage nodes in a scalable way (proposal WIP)

## Video packaging (manifest generation & delivery)
There are two options available for manifest generation & then storing the manifest files:
1. After transcoding the media file into multiple resolutions/bitrate streams, Orion can generate the DASH/HLS manifest files and upload these files too as data objects to the storage node. Then for playback, clients can request the manifest files from the distributor node.
2. Whenever the video playback is requested, Orion can generate the manifest file on the fly based on the data object streams available of different bitrates for the video. And then return the manifest file to the client.

Currently, Orion's Graphql video entity query returns the `resolvedUrls` object which contains the list of the absolute paths for the media assets, however for ABR streaming integration the query would return the manifest file path in the `resolvedUrls` object.

```gql
query videoById(id: "{videoId}") {
media {
id
# Returns the absolute path for manifest file
# e.g. "https://distribution-node-1/api/v1/assets/{mainfestId}" (if the manifest file is persisted)
# OR. "https://orion.joystream.org/api/manifest/{mainfestId}" (if manifest is generated on the fly & is served by Orion)
resolvedUrls
}
}
```

Using the first approach there is an extra cost associated with storing the manifest file. On the other hand, the second approach could delay playback as Orion would need to perform packaging & generate the manifest file, However, we could cache the manifest file on the server side (Orion) so the subsequent queries for a given video would be resolved instantly.
Also, keep in mind, even if we go with the first approach, the manifest file still needs to be validated (i.e. ensure that all the distribution -nodes specified in the `` tag are still serving the channel bag, or the channel hasn't removed/replaced any assets that are part of the manifest file `` tag, etc.). So I am inclined towards using the second approach.

# Tooling Available

## Transcoding
- ffmpeg(https://github.com/FFmpeg/FFmpeg)

## Packaging
For a manifest generation, the following tools are available
- shaka-packager(https://github.com/shaka-project/shaka-packager). It can generate both DASH & HLS manifests
- Bento4 (https://github.com/axiomatic-systems/Bento4)
- GPAC (https://github.com/gpac/gpac)

## Browser players/clients available for ABR
- [video.js](https://github.com/videojs) (Atlas currently uses this player, and does not support Adaptive bitrate streaming out of the box, however, it has DASH & HLS plugins available to support ABR)
- [dash.js](https://github.com/Dash-Industry-Forum/dash.js/) - A reference implementation from DASH-Industry-Forum
- [hls.js](https://github.com/video-dev/hls.js/)
- [shaka-player](https://github.com/shaka-project/shaka-player) (supports playing both HLS & DASH streams)

# Optimising media streams for ABR
- Encode multiple bitrates/resolutions of the same content. Here are the bitrate recommendations from [Youtube](https://support.google.com/youtube/answer/1722171?hl=en#zippy=%2Cbitrate) for different resolutions. I think based on this recommendation uploading 4 transcoded assets (from 360p to 1080p) is enough to start with. So a total of 5 (1 audio, 4 videos) assets. WDYT?
- Jump/Difference between bitrates should be consistent & minimum so It is easier to switch between different resolutions when network throughput change.
- Recommended GOP size should be [5s](https://github.com/shaka-project/shaka-packager/blob/main/docs/source/tutorials/encoding.rst#general-guidelines-of-how-contents-should-be-encoded) or less. As the streams are usually switchable only at GOP boundaries. A smaller GOP size results in faster switching when the network condition changes.