THU-MAIC / THU-MAIC/OpenMAIC

[Feature]: Video & External Media Embedding

Open
#203 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
TypeScript
Stars
37.2k
Forks
5.9k
Avg merge
1d 3h
Merged PRs (30d)
195

Description

### Problem or Motivation

Currently, scenes can only contain AI-generated images, text, and interactive HTML simulations. There is no way to embed external media — specifically video links (YouTube, Vimeo, Loom), external URLs, or iframe-based content.
This is a significant limitation for real-world educational use cases. When I create a classroom about renewable energy, I want to embed a 2-minute YouTube video showing how a wind turbine is manufactured, link to a live interactive map of energy cooperatives across Europe, include a Loom walkthrough of a software platform, or reference a TED talk as supplementary material within the lesson flow.
Right now, the only workaround is to mention the URL in the narration text ("go visit this link..."), which breaks the immersive classroom experience. The learner has to leave the classroom, open another tab, watch the video, and come back — losing context and flow.
Video is the most requested content type in online learning. Without it, OpenMAIC classrooms feel like "slides with voice" instead of the rich multimedia experience they could be.

### Proposed Solution

2.1 — Video Embed Scene Element
Add a new element type to slides: video-embed. It should support:

YouTube / Vimeo links — auto-detected and rendered as embedded players (using standard iframe embed URLs)
Direct video URLs — MP4, WebM files hosted externally
Uploaded videos — user uploads a video during classroom creation, stored alongside other media
Playback controls — play/pause, volume, fullscreen, timestamp seeking
Auto-pause narration — when a video starts playing, the AI narration pauses and resumes after the video ends (or after a configurable duration)

2.2 — External Link Cards
Add a clickable link card element that renders as a rich preview (title, description, thumbnail) for external URLs. Similar to how Slack or Discord preview links. Useful for referencing articles, documentation, tools, or live demos without leaving the classroom.
2.3 — Generation Support
During lesson generation, the AI should be able to:

Suggest relevant YouTube videos based on the topic (using web search)
Accept user-provided video URLs in the input prompt ("include this video on slide 3: https://...")
Place video embeds at appropriate points in the lesson flow (e.g., after explaining a concept, before a quiz)

2.4 — Export Handling

PPTX export: Insert a clickable thumbnail image with the video URL as a hyperlink (PowerPoint doesn't support inline video from URLs)
HTML export: Embed the video player directly in the exported HTML page
Classroom playback: Native inline playback with narration synchronization

Technical suggestion
The lib/action/ engine already handles 28+ action types. Adding a video-play action type that takes a URL, duration, and autoplay flag would fit naturally into the existing architecture. The components/slide-renderer/ would need a new VideoElement component wrapping an iframe or HTML5 video player.

### Alternatives Considered

Interactive Simulation workaround: I tried embedding a video via the HTML simulation feature by writing custom HTML with an iframe. It partially works but is hacky — no narration sync, no proper sizing, and it doesn't appear in PPTX export.
Text link in narration: Mentioning "visit youtube.com/watch?v=..." in the spoken narration. Very poor UX — breaks immersion, users have to manually navigate.
Upload video as "document" input: Uploading a video file as source material for generation. OpenMAIC parses documents but ignores video files entirely.

None of these provide a native, integrated video experience within the classroom flow.

### Area

Classroom generation

### Additional Context

Use case examples:

Corporate training: "Watch this 90-second safety video, then answer the quiz" — video must be inline, not a separate link
Science education: Embed a physics experiment recording between the theory slide and the interactive simulation
Product demos: Embed a Loom screen recording showing how to use a software tool
Language learning: Embed a native speaker video clip before a pronunciation exercise

Supported platforms to prioritize:

YouTube (most common)
Vimeo (professional/corporate)
Direct MP4/WebM URLs
Loom (screen recordings)
Bilibili (for Chinese users, aligning with existing zh-CN support)

Narration sync is the key differentiator. Any slide tool can link to a video. What makes this valuable in OpenMAIC is the AI teacher saying "Now let's watch a short clip about..." → video plays → AI teacher resumes "As you just saw in the video..." — this creates a seamless classroom experience that no other tool offers.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.