cloudwego / cloudwego/eino-ext

Native Video Generation Model Support

Open
#951 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
811
Forks
368
Avg merge
16h 22m
Merged PRs (30d)
13

Description

## Description

Add native **Video Generation Model** support to Eino, following the same component pattern currently used for `ImageGenerationModel`
/
Developers should be able to configure a supported video generation provider/model, pass natural-language prompts or multimodal inputs, and receive generated video outputs through a standardized Eino interface.

The goal is to make video generation a first-class model capability alongside chat and image generation.

### Proposed capabilities

* **Text-to-video** — generate a video from a text prompt.
* **Image-to-video** — provide an image as the starting frame/reference and animate it using a prompt.
* **Video-to-video** — where supported by the underlying provider.
* Configure generation parameters such as:

* duration
* resolution
* aspect ratio
* FPS
* seed
* model/provider-specific options
* Support video outputs as **URL, file, or binary data**, depending on the provider.
* Support asynchronous generation for providers where video generation is submitted as a job and completed later.
* Expose generation/job status such as `queued`, `processing`, `completed`, and `failed`.
* Preserve provider metadata such as request ID, task ID, model, duration, and generation parameters.
* Provide a consistent interface so additional video-generation providers can be implemented without changing application-level agent code.

### Example API direction

The developer experience could follow the existing image generation API:

`VideoGenerationModel.Generate(ctx, messages)`

For asynchronous providers, Eino could additionally expose a task/job abstraction:

`Generate -> Task ID -> GetStatus / Wait -> Video Output`

This would allow Eino agents to treat video generation as a native multimodal capability rather than requiring developers to implement provider-specific APIs and polling logic themselves.

## Use Case

An Eino agent could receive a request such as:

> "Create a 10-second 16:9 cinematic video of a spacecraft landing on Mars."

The agent could invoke the configured video generation model, monitor the generation job, and return the resulting video artifact to the user.

This would bring video generation to the same abstraction layer as Eino's existing LLM and image-generation components.

Contributor guide

Open the contributing guide

Research direction

Start by reading Eino's existing ImageGenerationModel component and its Generate(ctx, messages) interface to understand the current model pattern. Compare the requested text-to-video, multimodal input, asynchronous task status, output forms, and provider metadata against that pattern. Done means a settled, provider-independent video-generation design with clear synchronous and asynchronous behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.