meta-pytorch / meta-pytorch/torchcodec
Legacy `sws_scale()` prevents multi-threaded color conversion
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.2k
- Forks
- 125
- Avg merge
- 22h 47m
- Merged PRs (30d)
- 54
Description
🚀 The feature
Following up on #1215, the current implementation of sws_scale is a bottleneck when decoding 4K videos. On the plot below, you can see that decoding is capped at ~20ms regardless of the number of threads and seeking mode of the decoder (CPU decoding).
After some investigation, it turns out the YUV -> RGB conversion is dominating the decoding time and is very consistent. torchcodec uses the legacy sws_scale() FFmpeg function for YUV→RGB color conversion. This function is always single-threaded. FFmpeg 7+ introduced sws_scale_frame() with a threads option on SwsContext that parallelizes the conversion across CPU cores. PyAV and the ffmpeg CLI already use this new API, achieving roughly 2x faster conversion on 4K content.
Motivation, pitch
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating torchcodec’s use of the legacy sws_scale() path and reading how its SwsContext is configured. Compare that path with FFmpeg 7+ sws_scale_frame() and its threads option; done means YUV→RGB conversion can use multiple CPU threads and avoids the reported 4K decoding bottleneck.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- audio-video-rtc, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100