In mobile browsers, seamlessly switch video (with audio) to audio-only on screen lock/when backgrounded
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 16.9k
- Forks
- 2.8k
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 27
Description
Is your feature request related to a problem? Please describe.
(Apologies if this isn't flushed out enough to belong under Feature Request.)
I have a use case that I think is becoming very common on mobile devices. Consider video + audio media where audio is the priority - like a podcast, a DJ playing music, or playback of a music video. When the user puts the video out of view, rather than pausing video and audio (as is standard for all mobile browsers), it would be really great to seamlessly transition to audio only and continue playing. When the user puts the video element back into view, the video picks up at the same time as the audio. On the surface, this is what the native Spotify Android and iOS mobile apps have done for video podcasts (although they may just keep playing the video underneath). I haven't found a good way to achieve this in mobile browsers aside from some hacks, like playing audio in an <audio> tag and audio-less video in a <video> tag at the same time. This works quite well at first - it's not difficult to wire up custom controls and keep them both in sync (event handlers do a good job here). It's also nice that when the video is out of view and paused by the system, we can easily stop it from downloading and bandwidth is saved. But this hack quickly starts to fall apart the moment external controls or external viewing is used - like the notification panel Play/Pause actions (especially on iOS), picture in picture controls, or chromecast. The 2 separate media elements being played are unexpected and not handled well.
Describe the solution you'd like
I would like to consider using Hls.js to remove video on-the-fly and in a way that works with current mobile browser behavior and allows a video's audio to continue playing when the video element is put out of view, achieving a similar experience in a mobile browser that one has when playing a video podcast in a native Spotify mobile app. A basic approach could be to disable video (at the demuxing or the SourceBuffer stage for example). The video track is simply being enabled and disabled at the right time. Any approach taken would seem to require some kind of "onBeforeVideoPausedBySystem" event in order to remove the video "in time" for the audio to continue seamlessly. (Although this may not be reliable and may be the wrong approach.)
Describe alternatives you've considered
I realize this may be better solved by the browser (in Chromium, Webkit, etc) and may even require it (considering the browser behavior that pauses video when it goes out of view is not controllable and may not be directly observable from JS land). It could be a relatively simple <video> tag attribute/property that achieves "continue playing audio when system pauses video because it is out of view" and abstracted away from the implementation layer. That being said, there are details about behavior that one could want control over.
UPDATE: I see on-the-fly video to audio-only switching is covered a bit here - https://github.com/video-dev/hls.js/issues/163. Curious if things have progressed far enough to achieve this kind of thing.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing related issue #163 and the browser behavior described for screen lock, backgrounding, Picture-in-Picture, external controls, and Chromecast. No source files or tests are named; a useful outcome would be a settled feasibility and implementation scope for seamless video-track removal while audio continues, including how completion could be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- html, javascript, typescript
- Domain
- audio-video-rtc
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100