THU-MAIC / THU-MAIC/OpenMAIC

Normalize image/video generation constraints per provider and model

Open
#527 0 comments 0 reactions 0 assignees View on GitHub
area:generation area:providers priority:P2 status:icebox type:task
Dominant language
TypeScript
Stars
37.2k
Forks
5.9k
Avg merge
1d 3h
Merged PRs (30d)
195

Description

## Context

A runtime OpenAI image generation error exposed a broader media pipeline issue:

`OpenAI image generation failed (400): Invalid size '1024x576'. Requested resolution is below the current minimum pixel budget.`

The immediate failing case was OpenAI, but the root problem is not provider-specific. Today we often convert `aspectRatio` into generic `width/height` or pass coarse registry values before validating against the selected provider/model's actual API constraints.

## Problem

Media generation constraints are currently scattered across:

- provider registry metadata
- generic helpers like `aspectRatioToDimensions`
- adapter-specific request shaping
- implicit provider API defaults

This makes it easy to send invalid requests, especially when constraints differ by model rather than provider.

Examples from docs:

- OpenAI `gpt-image-2` supports arbitrary `size`, but only within pixel/edge/ratio constraints.
- Seedream size limits differ between 3.0, 4.0, and 4.5.
- Qwen Image 2.0 and Qwen Image Max/Plus use different allowed size strategies.
- MiniMax Image supports both `aspect_ratio` and `width/height`, but model behavior differs.
- xAI image/video supports native `aspect_ratio` / `resolution`, but adapters do not fully pass them today.
- Video providers have model-specific duration/resolution matrices, especially Veo, MiniMax, and Seedance.

## Proposal

Add a model-aware media capability normalization layer before adapter execution.

### Scope

1. Introduce a central capability registry for image and video generation keyed by `providerId + modelId`.
2. Represent provider/model constraints explicitly, for example:
- image fixed size presets
- image arbitrary size constraints
- native aspect ratio support
- native resolution support
- minimum/maximum total pixels
- edge divisibility requirements
- supported video durations
- supported video resolutions
- valid duration/resolution combinations
3. Normalize user/request options before calling adapters.
4. Ensure adapters receive already-normalized options and only handle provider-specific wire format mapping.
5. Log both requested and normalized media parameters for easier debugging.
6. Add tests covering at least:
- OpenAI `gpt-image-2` invalid low-pixel 16:9 normalization
- Qwen fixed-preset vs arbitrary-size behavior
- Seedream model-specific size constraints
- MiniMax Image aspect ratio and width/height precedence
- xAI image/video aspect ratio and resolution pass-through
- Veo duration/resolution/aspect ratio combinations
- Seedance and MiniMax video duration/resolution matrices

## Non-goals

- Adding new providers
- Changing provider credentials/config fallback behavior
- Reworking UI flows beyond reflecting normalized capabilities where needed
- Solving Kling completely until official docs are manually verified

## References

- OpenAI image generation: https://developers.openai.com/api/docs/guides/image-generation
- Volcengine Seedream: https://www.volcengine.com/docs/6492/2172373
- Volcengine Seedance: https://www.volcengine.com/docs/6492/2165104
- Alibaba Qwen Image: https://help.aliyun.com/zh/model-studio/qwen-image-api
- Google Nano Banana: https://ai.google.dev/gemini-api/docs/image-generation
- Google Veo: https://ai.google.dev/gemini-api/docs/video
- MiniMax Image: https://platform.minimaxi.com/docs/api-reference/image-generation-t2i
- MiniMax Video: https://platform.minimaxi.com/docs/api-reference/video-generation-t2v
- xAI Image: https://docs.x.ai/developers/model-capabilities/images/generation
- xAI Video: https://docs.x.ai/developers/model-capabilities/video/generation

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.