Comfy-Org / Comfy-Org/ComfyUI

Feature Request: World Lab API

Open
#12,802 3 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
133k
Forks
15.7k
Avg merge
1d 6h
Merged PRs (30d)
155

Description

### Feature Idea

https://www.worldlabs.ai/blog/announcing-the-world-api

Somr initial ideas dumping:

**1. "Image-to-World" Custom Nodes**
The most direct integration would be a set of nodes that wrap the World API endpoints.

The Workflow: An artist generates a high-quality 2D environment using a standard Stable Diffusion/Flux workflow in ComfyUI. The output image is then passed into a "WorldLabs: Generate World" node.

The Benefit: This allows creators to iterate on the aesthetic of a scene in 2D (lighting, style, mood) and then "instantiate" it into a navigable 3D space with a single click.

**2. Multi-View & Panorama Stitching**
The World API supports multi-view inputs and 360° panoramas.

The Workflow: ComfyUI is excellent at batch-processing. You could use a workflow to generate 4–6 consistent views of a room (using ControlNet to maintain layout) and pipe them into a "WorldLabs: Multi-Image World" node.

The Benefit: This would produce much higher-fidelity 3D reconstructions than a single image, as ComfyUI can precisely manage the spatial consistency of the input images before they hit the World API.

**3. "World-to-Video" Motion Pipelines**
One of the most exciting potential workflows is using the generated world as a foundation for video.

The Workflow: 1. Generate a 3D world via World API.
2. Use a "WorldLabs: Virtual Camera" node to define a path (spline) through that 3D world.
3. Export those frames back into ComfyUI for AnimateDiff or SVD (Stable Video Diffusion) to add "micro-movements" like flowing water or moving clouds.

The Benefit: It solves the "temporal consistency" problem in AI video by grounding the movement in a persistent 3D geometry.

**4. Interactive Pre-Visualization for Filmmakers**
Targeting the professional VFX and indie film community (a key demographic for Comfy.org):

The Idea: A "Live Link" node. As you adjust your text prompt or LoRA strengths in ComfyUI, a low-resolution preview of the 3D world is streamed back into a side panel.

Use Case: A director can "block" a scene by generating a 3D world from a sketch, then use the World API's exported Gaussian Splat or Mesh files directly in Unreal Engine or Blender via the ComfyUI-to-3D pipeline.

### Existing Solutions

_No response_

### Other

_No response_

Contributor guide

Open the contributing guide

Research direction

The issue links the World API announcement and describes possible ComfyUI nodes for image-to-world, multi-view, video, and virtual-camera workflows, but names no repository files or tests. Start by reading the linked API announcement and reviewing ComfyUI's existing node and external-service integration entry points. Done criteria need an agreed first workflow, API/authentication details, and acceptance tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
blender, python, unreal-engine
Domain
ai, api, computer-graphics
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.