FFmpegInterop really struggles to play some 1080p videos and all 4k videos

iPhones now captures videos in 1080p and 4k and as a result they have become so common. I have my own uwp video project where I'm trying to play such videos. I had lot of issues with framerate, that's why I checked this project.

1. First of all, audio and video pipeline are not split up into separate threads in ffmpeg interop. That way, cores can be better utilized. Also, decoding and scaling the frame too. This made 1080p videos play much smoother, but 4k was still pretty poor.
2. Then I able to solve most of the frame rate issues by asking FFMPEG to multi-thread the decode process (which was also recently added by FFmpegInterop). But the problem now seems to be sws_scale. It takes ~30ms on my machine (4 core 8 thread - i7) to scale one frame.
   Note: I'm only talking about the case where Ffmpeg does the decoding and not Windows.

My question is: Is there any way by which sws_scale can be avoided all together and somehow we could render the YUV frame onto the screen directly? I feel that's the only way we could keep up the frame rate. VLC Store app is somehow able to do a fantastic job, even using less CPU. I tried this both after turning off H/W accl and turning on H/W accl.

Start by tracing the FFmpeg decode pipeline and the sws_scale step described in the issue, including how audio and video processing are coordinated. Determine whether decoded YUV frames can be rendered directly, then validate the result against the reported 1080p and 4K frame-rate problems.