Comfy-Org / Comfy-Org/ComfyUI_frontend
[Feature Request]: (BIG feature) Realtime interactive viewport (image view window)?
- Dominant language
- TypeScript
- Stars
- 2k
- Forks
- 699
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 495
Description
### Is there an existing issue for this?
- [x] I have searched the existing issues and checked the recent builds/commits
### TL;DR
If Comfy wants to ever get **convenient** and truly **interactive** image editing nodes - whether built-in or third-party ones - it needs to implement a standard window to draw image output, which also provides an **easy** addition of custom controls for node developers.
----------
### The issue in detail
Most (if not all) image editing software has a clear distinction: you **work with the graph** in one panel/window, but you **view the output** of a node in the other. Comfy, however, goes in its design against this widespread convention, and instead pushes the idea of preview nodes, which has a few significant UX flaws:
- You can't view an output of a node from one side of the graph while tweaking parameters on the other;
- You constantly need to zoom in and out to view an image in detail _(for the first time in decade, I have my mouse wheel literally broken - solely because of Comfy)_;
- Unless you use special comparison nodes, it's very hard to compare images from the different stages of the workflow: it's literally easier to open images in browser tabs _(in other words, the best solution in the current design paradigm is doing it OUTSIDE this paradigm)_;
- You can't zoom in really close unless you make the preview node bigger, which eats up already precious real estate;
- Since each preview node has its own image and many of them could be shown at once, each such node in the graph gradually slows the browser, even when it's muted (especially with ultra-HD images); moving an entire big graph with many such nodes - just forget about it;
- And last but not least: without a preview window, **there's also no standard way of doing anything interactively ON the image** - providing immediate visual feedback right in the browser.
The issue boils down to a core design decision. Since there's no **standard** window to show the selected node's output, each node has to do it on it's own from scratch, if it needs to show ANYTHING interactively, and those that do, implement this interactive interface purely with JS, which is ridiculosly slow (try it yourself with ControlNet skeleton editors, while having 5-6 skeletons on the image).
And here we end up with a chicken and egg problem: by itself, such a preview window aka viewport (as it's called in other software) has next to no benefit to a user (compared to the current approach), so it seems not worth implementing. But **without** it - in addition to the listed UX issues, there's also a plethora of nodes that will **never** be developed since it's too much of a work. Basically anything that needs providing an interactive **visual** feedback becomes a tremendous task - involving a skillset necessary for coding the actual node in python PLUS a skillset to develop JS interfaces in browser. The simple example of such a node is a node to draw vector shapes.
## What would your feature do ?
Sooo... maybe ComfyUI has matured enough to consider a more standard approach to displaying image stuff? Not a special preview node, but a special **window**, which always stays in the same place and changes it's **contents** depending on what node is active?
## Proposed workflow
### MVP
- Implement a viewport window/panel to show output image of a single node.
- In terms of layout, it could be hard-coded to always be on the left/right of the graph.
- The viewport can be enabled/disabled in settings and users can stick to whatever they prefer: the current preview-nodes system, the viewport, or even both.
- The viewport can't do anything other than showing the image and letting a user to zoom/pan it as a canvas (the same way node graph does).
- Only a single output of a single active node can be displayed at once.
- A simple dropdown should be in the window to choose which of the image outputs to show (if a node has multiple), defaulting to the first one found.
- When switching between the nodes, the relative zoom should be preserved, as well as the active output (if both nodes have an output with the same name).
- The bar that appears on node selection needs a new button added - to make the node active for viewport.
- Additionally, Nuke/Fusion approach (to node activation) can be adopted: you can make multiple nodes semi-active by selecting them and hitting 1-9 buttons on the keyboard. Then, if mouse is over viewport, hitting the same number key switches to displaying the corresponding "viewport input".
- This is **VERY** convenient for comparison or viewing how various masks/filters affect the final image. A number could be shown in place of the "activate" button on the node - to indicate which "viewport input" it takes. Hitting the same number with other node selected switches this input to it.
### Further improvements
- A toggle in the viewport to auto-activate any selected node which has an image output.
- The viewport is capable of showing BOTH image and mask.
- A toggle to enable/disable alpha display for images which have it, or to show alpha only (as grayscale image).
- The viewport can be freely docked as a panel on any of four sides of the canvas.
- A toggle to enable/disable filtering on zoom (it's often useful to see the raw pixels up close).
- Buttons/hotkeys for a few common zoom states: 1:1 scale centered, fit the entire image into the window centered, center the image preserving the zoom, etc.
### The long term goal - interactive-preview-API
Even though the MVP version is intended as a display only, the viewport needs to be designed (from it's inception) as a thing which is **supposed** to have a custom interactive interface provided by a node. Various custom buttons, gizmos drawn on top of the image, etc.
In the future, I'd like to hope there would be some API allowing nodes to add some custom controls and link them with their properties. So a node would have special methods specifying the in-viewport buttons a node has, custom tools a user can activate with those buttons, etc. Those tools are nothing more than a user-friendly way of setting node parameters. Instead of nudging a slider which specifies where to position a square or how much to turn it, a user simply drags this square on the image.
And as a pinnacle of this feature, there should be a standard approach for:
- Making a node which produces its output entirely with a GLSL shader - so it can do it interactively in the browser, fully on the client side with WebGL, without hitting the "Run" button, but truly immediately - as user tweaks parameters. The same GLSL shader is used in the backend when the node gets evaluated there.
- Using the same GLSL/WebGL combo, a node could draw ANY custom overlay or an image effect. Basically, same as the previous one, but as a second step only in frontend. So, the contents of the viewport with such node active would do the following:
- Input image is processed with the main GLSL shader (the one that implements the node behavior)
- The result is processed with the second shader which does overlay and may draw some masks, gizmos, zone outlines, etc.
- The layer of buttons/controls defined with the viewport API is drawn on top.
Yes, "shader" is a scary word which sounds like some out-of-reach rocket science for most node developers, but SOME would be able to implement extremely user-friendly and fully interactive interfaces capable of showing ANYTHING, completely on the client side and with a MUCH better performance than processing images with libraries like NumPy or even PyTorch _(processing images with native graphics API is always better optimized than arbitrary code execution with GPGPU)_.
### Additional information
_No response_
┆Issue is synchronized with this [Notion page](https://www.notion.so/Issue-4342-Feature-Request-BIG-feature-Realtime-interactive-viewport-image-view-window-2256d73d365081aaabbde6d69c2d7c0a) by [Unito](https://www.unito.io)
Contributor guide
Assessment
This issue has not been assessed yet.