Folder image list processing is inefficient for long batch workflows; consider image-by-image streaming execution
- Dominant language
- Python
- Stars
- 133k
- Forks
- 15.7k
- Avg merge
- 1d 10h
- Merged PRs (30d)
- 153
Description
### Problem
Using the built-in **Load Image from Folder** node for a large set of images appears to execute the workflow in a list-oriented, node-by-node manner rather than letting each image run through the entire graph and be saved before moving to the next image.
For batch transformation workflows (for example, upscaling), this has several practical downsides:
- noticeably worse throughput than queueing images one at a time
- intermediate image data can stay alive longer than necessary, increasing RAM pressure
- output is not written as early as it could be
- if a downstream node errors late in the graph, a large amount of already-computed work may be lost and must be repeated
In one real workflow, processing about **150 images** via the folder loader took **more than 30 minutes**, substantially slower than expected compared with per-image execution.
### Desired behavior
For folder/dataset-style image processing, it would be useful to have an execution mode that behaves like a streaming `for each image` pipeline:
1. load one image
2. run that image through the full graph
3. save/output it
4. release intermediates where possible
5. continue with the next image
This would improve throughput, memory behavior, failure isolation, and resumability for large image-processing jobs.
### Request
Please consider either:
- changing the folder-loader execution path to support image-by-image streaming through the full graph, or
- adding an explicit execution mode/node for this behavior.
The current list semantics are useful in some workflows, but they are a poor fit for large batch conversion/upscaling jobs.
Contributor guide
Research direction
Start by tracing the built-in Load Image from Folder node and its list-oriented execution path, then compare it with queueing images one at a time. Define how a streaming mode would load, process, save, and release one image before continuing, while preserving the current list semantics where needed. Done means large folder workflows improve throughput, memory behavior, failure isolation, and resumability.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100