Genymobile / Genymobile/scrcpy

MJPEG stream option: trade jpeg artifacts for crispiness and temporal stability

Open
#6,340 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
150k
Forks
13.7k
Avg merge
4d 13h
Merged PRs (30d)
2

Description

- [X] I have checked that a similar feature request does not already exist.

#3750 is perhaps asking for exactly the same thing? but they were incorrectly shut down and the issue is closed now.
#3680 is discussing the client reencoding for further network broadcast?

**Context**
I have a cheap ``HDMI`` -> ``High-Speed USB 2.0`` capture card. The way it does 1080p at reasonable framerates is by encoding video as MJPEG, literally all keyframes are JPEGs. And the quality of still frame I get from it is leagues ahead of anything I can possible hope to get from scrcpy.

Phone-as-a-webcam apps like [DroidCam](https://play.google.com/store/apps/details?id=com.dev47apps.droidcam)/[X](https://play.google.com/store/apps/details?id=com.dev47apps.droidcamx) provide MJPEG sctream since forever ago, I ran older version of DroidCam for android 2.6.x and both the encoding performance and battery life were satisfactory even back then, and are perfectly fine now with 1080p.

I don't know exaclty how they are encoding it on android, seems to me this would require software encoder like libjpeg-turbo and fairly simple bundling of those frames as video. MJPEG streams over HTTP are literally a sequence of JPEGs separated by a boundary string, disguised as a single HTTP file download. All they do is announce the boundary as part of ``content-type`` HTTP header: ``content-type
multipart/x-mixed-replace;boundary=--dcmjpeg`` and then keep pushing data.

**Is your feature request related to a problem? Please describe.**
Every single video encoding option available on my phone can only encode the entire screen in such a way a pixel block is (perhaps indirectly) dependant on all other blocks. Any change of any pixel acts like source of prng noise for the whole frame. This means there is absolutely no escaping of ``heavy noise when keyframe interval is 0`` or ``heavy quality loss when the entre screen is moving/scrolling``. Pretty much, no temporal stability no matter what I do.

**Describe the solution you'd like**
JPEG is a lossy format that does block-to-block independent encoding *by default*. You set target quality and can literally see individual 8x8 blocks getting extra jpeg artifacting noise from movent. However! All the other blocks on the entire screen are staying perfectly stable frame to frame given the pixels they cover don't change.

**Describe alternatives you've considered**
Setting frame interval to 0 and [bitrate mode](https://developer.android.com/reference/android/media/MediaFormat#KEY_BITRATE_MODE) to [CBR](https://developer.android.com/reference/android/media/MediaCodecInfo.EncoderCapabilities#BITRATE_MODE_CBR) encoding: ``--video-codec-options=i-frame-interval:int=0,bitrate-mode:int=2``
This seems to be the default anyway as I could not see improvement.

**Alternatively, a.k.a. maybe I'm dumb**
Perhaps there are better ways to ahieve all keyframes stream with block-to-block independant encoding? So much time has passed since JPEG, people found [simpler ways](https://qoiformat.org/) to do lossless.

Perhaps other formats could work better for a similar goal. [ASTC with 6x6](https://github.com/ARM-software/astc-encoder/blob/main/Docs/FormatOverview.md#block-sizes) blocks should easily fit 1080p60 stream into usb2.0? I haven't put that much time researching newer formats, but the goal here is not to do JPEG per se, it's to get better results with desired tradeoffs.

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files or tests are named. Start by reviewing the related issues #3750 and #3680, then inspect the Android MediaCodec options and the proposed MJPEG multipart stream format. Done requires a decided encoding approach, documented tradeoffs, and an agreed implementation and validation plan.

Written by the indexing model from the issue text.

Assessment

Tech stack
android
Domain
audio-video-rtc, mobile-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.