google / google/knusperli

[REQ] MPEG (1/2/4) video decoding

Open
#19 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
474
Forks
27
PR merge metrics
No merged PRs in 30d

Description

Hi there,
just discovered this interesting approach to JPEG decoding.

I've just asked GH-Copilot (Claude Haiku 4.5) if the technique is applicable to video decoding too:

> Based on my review of the Knusperli repository, here's my analysis of whether its approach can be applied to MPEG-encoded videos:
>
> ## Short Answer: **Yes, but with significant practical challenges.**
>
> ## How Knusperli Works (JPEG-focused)
>
> Knusperli reduces blocking artifacts in JPEG images by:
> 1. Treating quantized DCT coefficients as **intervals** rather than fixed values
> 2. Selecting values within those intervals that **minimize discontinuities at 8×8 block boundaries**
> 3. Maintaining mathematical equivalence (coefficients still round to the stored quantized values)
>
> ## Applicability to MPEG Video
>
> ### ✅ **Theoretically Compatible**
> - **MPEG-1/2/4 video codecs** also use DCT-based block coding (typically 8×8 blocks)
> - **Similar quantization scheme**: MPEG also quantizes DCT coefficients and discards precision
> - The **core algorithm** (interval-based coefficient selection to reduce blocking) could apply frame-by-frame
>
> ### ⚠️ **Major Practical Challenges**
>
> 1. **Temporal Coherence**: MPEG uses motion compensation and inter-frame prediction. Knusperli's block-boundary smoothing could introduce:
> - Temporal flickering between frames
> - Conflicts with motion vectors
> - Artifacts at frame boundaries in predictive coding
>
> 2. **Computational Complexity**:
> - Videos have **orders of magnitude more frames** than a single image
> - Real-time or near-real-time processing becomes critical
> - Knusperli's optimization per frame would need to be extremely efficient
>
> 3. **Chroma/Motion Components**:
> - MPEG's chroma subsampling (4:2:0) differs from standard JPEG handling
> - Motion prediction adds another layer of quantization that Knusperli doesn't address
>
> 4. **Modern Codec Differences**:
> - H.264/H.265/VP9/AV1 use more sophisticated entropy coding (not just Huffman)
> - Larger variable block sizes (beyond 8×8)
> - Would require substantial adaptation
>
> ### 🔄 **Potential Adaptations**
>
> 1. **Frame-by-frame processing** with temporal consistency constraints
> 2. **Motion-aware deblocking** that respects inter-frame predictions
> 3. **Selective application** to I-frames only (less risky temporally)
> 4. **Integration with existing post-processing** rather than standalone approach
>
> ## Recommendation
>
> This would be a worthwhile research project, but you'd need to:
> - Benchmark temporal stability across frame sequences
> - Design motion-aware constraints
> - Test on various MPEG versions/profiles
> - Consider modern codecs (H.264/HEVC) which already have built-in deblocking filters
>
> The technique is **not a drop-in replacement** but could inspire a **video-specific deblocking enhancement**.

Source: https://github.com/copilot/share/4a4f02b4-4920-84b5-a843-8a09e008080a

...it would be interesting to challenge Gemini for coding this...

Hope that inspires.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.