intel / intel/AI-Playground

[Feature Request] Real-Time "System-Audio" Translation & Overlay (Omni-Translate)

Open
#465 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
979
Forks
132
Avg merge
9h 23m
Merged PRs (30d)
8

Description

I am frequently frustrated by the poor quality of "Live Captions" and "Auto-Translate" on platforms like YouTube and Nico Live. When watching Japanese livestreams or international technical briefings, the built-in translation is often too literal and lacks nuance. It frequently fails on technical hardware terminology or regional slang (e.g., Japanese music scenes).

While the current AI Playground 3.1.0a release features excellent local LLM support, there is no streamlined way to apply that intelligence to real-time video media playing in a browser.

I would like to propose a "Live Omni-Translate" feature for AI Playground Studio that provides high-fidelity, real-time transcription and translation via a local on-screen overlay.

Proposed Workflow?

System Audio Loopback: Implement an option to capture audio from the Windows system/browser (WASAPI) instead of just the microphone.

Two-Stage Pipeline?

Transcription: Use OpenVINO-Whisper (v3) for low-latency, real-time speech-to-text.

Contextual Translation: Pipe the raw text through a local LLM (specifically the newly supported Gemma 4-26B MoE) with a specialized system prompt. This would provide the natural, "ChatGPT-level" translation quality that standard lookup-based tools lack.

Real-Time Overlay: A transparent, "always-on-top" UI element that renders the translated text directly over the video player (YouTube, Nico Live, etc.).

Alternatives I've tried:

Browser extensions: Some have a timed limit, character translation limit, you have to pay for them after a trial, etc.
Platform Native Captions: High error rate and poor handling of niche gaming/tech jargon.
Cloud-based Browser Extensions: Most require paid API keys (DeepL/Google), which introduces privacy concerns and monthly costs.
Current 3.1.0a Voice Mode: It works great for mic input, but lacks the system-audio capture and overlay rendering needed for media consumption.

This feature would be a "hero use-case" for Intel HW: The NPU could handle the Whisper STT transcription and the Arc GPU could simultaneously run the Gemma 4-26B MoE model for the translation layer.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.