Comfy-Org / Comfy-Org/comfy-action
Add assertion inputs so a green run means "the workflow produced the right output", not just "the process exited 0"
- Dominant language
- Python
- Stars
- 36
- Forks
- 13
- PR merge metrics
- No merged PRs in 30d
Description
### Summary
`comfy-action` currently passes a workflow when `comfy run` exits 0 and at least one output filename can be scraped from its stdout. There is no input for *what the output should be*. That makes a green run compatible with several failure modes where the workflow silently did less than it was supposed to.
Comfy-Org already owns the missing piece — [`ComfyUI-test-framework`](https://github.com/Comfy-Org/ComfyUI-test-framework) ships assertion nodes and a `comfyci` CLI that exits 1 on assertion failure. This issue proposes wiring that (or an equivalent) into `comfy-action`, rather than inventing a new comparison format.
### 1. What the action accepts today
`action.yml` declares 16 inputs (lines 3–65). Grouped by what they control:
- **Environment:** `os`, `python_version`, `cuda_version`, `torch_version`, `specific_torch_install`, `comfyui_flags`, `use_prior_commit`, `timeout`, `skip_quickci`
- **Materials:** `models-json`, `workflow_filenames`, `workflow_raw_json` (marked `# Not yet supported`)
- **Where results are shipped:** `google_credentials`, `gcs_bucket_name`, `output_prefix`, `api_endpoint`
None of them describe the expected result. Outputs are uploaded to GCS and the CI dashboard, where a human looks at them.
### 2. Where pass/fail is actually decided
In `action.py`:
```python
result = subprocess.run(
["comfy", "--skip-prompt", "--no-enable-telemetry",
"run", "--workflow", file_path, "--timeout", "600"],
check=True, ...)
...
output_filenames = parse_raw_output(full_output)
if output_filenames is None:
if not os.path.exists(f"{args.workspace_path}/output/{args.output_file_prefix}_{counter:05}_.png"):
raise RuntimeError("Invalid output from Comfy-CLI, no outputs found")
```
So the pass criterion is **exit code 0 + "a file exists"**. Nothing reads the pixels.
(Side note in the same function: `parse_raw_output` keeps only the first line after `Outputs:` — `output = output[:outputnl]` — so a workflow with several output nodes only ever reports one file. Its own docstring already flags the text-scraping as `This is a hack`.)
### 3. Why exit 0 is weaker than it looks
ComfyUI validates **only the subgraph reachable from output nodes**. In `execution.py::validate_prompt`, output nodes are collected and then validated one by one:
```python
if hasattr(class_, 'OUTPUT_NODE') and class_.OUTPUT_NODE is True:
outputs.add(x)
...
for o in outputs:
m = await validate_inputs(prompt_id, prompt, o, validated)
```
Two consequences, both measured against ComfyUI 0.27.0 on 2026-09-03 with `--disable-all-custom-nodes` (stock nodes only):
**(a) A node that nothing consumes is never validated and never executed — silently.**
Posting `EmptyImage → PreviewImage` plus a disconnected node whose `width` is the string `"definitely-not-an-int"` and whose `batch_size` is `0`:
```
POST /prompt → 200
/history/ → status_str: "success", completed: true, outputs: ["2"]
```
No warning anywhere. A workflow can carry a completely broken node and CI stays green.
**(b) If one of two output branches fails validation, the prompt still succeeds.**
Same setup with a second `PreviewImage` fed by an `EmptyImage` whose `width` is invalid:
```
POST /prompt → 200, body contains node_errors: {"3": {... "invalid_input_type" ...}}
/history/ → status_str: "success", completed: true, outputs: ["2"]
```
`validate_prompt` returns success as long as `good_outputs` is non-empty; the failing branch is dropped with `logging.error("Output will be ignored")`.
And `comfy-cli` deliberately does not turn that into a failure — `comfy_cli/command/run/execution.py`:
```python
# 200 may still carry node_errors if some output chains failed
# validation but others passed — surface as warnings, not a failure.
node_errors = body.get("node_errors") if isinstance(body, dict) else None
self.validation_warnings = _node_errors_to_list(node_errors)
```
That is a reasonable default for interactive use. But in `comfy-action` it means **half a workflow can stop working and the job is still green**, because the exit code is 0 and one PNG exists.
(For completeness: a prompt with *no* output node at all is correctly rejected — `prompt_no_outputs`, HTTP 400. That case is already covered.)
### 4. Minimal proposal
Add inputs that let the caller state what the run should produce. Roughly, in order of increasing cost:
1. **`fail_on_validation_warnings`** (default `false`) — fail the job when `POST /prompt` came back with `node_errors`, i.e. when part of the workflow was dropped. This is the cheapest one and needs no new comparison logic.
2. **`expected_outputs`** — a path to a JSON/YAML file mapping each workflow to expected results, e.g. a perceptual hash plus a tolerance, and/or an expected output-file count. `ComfyUI-test-framework`'s `Assert Image Match` already implements dHash with a configurable `delta`; reusing its format would keep one baseline representation in the org.
3. **`test_workflows` / `use_comfyci`** — run `comfyci --server ...` instead of (or after) `comfy run`, and propagate its exit code. This gets assertions for images, masks, strings, tensor shapes and "was actually executed, not cached" for free, since those nodes already exist.
Option 1 alone would already close the case in §3(b). Options 2–3 are what turn the CI dashboard from "look at the pictures" into a regression gate.
### 5. Why I care about this specific gap
Separately from CI, I ran a controlled measurement on 2026-09-03 comparing two ways of rebuilding a workflow from stored generation parameters: for 112 records that one path reproduced faithfully, the other path returned **0 images that a human judged to be the same picture** — and 75 of those 112 failed *silently* (14 built nothing at all, 61 produced a different image). Nothing crashed; nothing exited non-zero. That is the shape of failure this ecosystem produces, and it is exactly the shape that "the process exited 0 and a PNG exists" cannot see.
Happy to open a PR for option 1 if the direction is acceptable.
### Environment for the reproductions
- ComfyUI 0.27.0, Windows portable build, `--cpu --disable-all-custom-nodes`, isolated SQLite DB, port 8199
- Measured 2026-09-03. Source line references checked against `comfyanonymous/ComfyUI@main` and `Comfy-Org/comfy-cli@main` on the same day.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with action.yml and action.py to trace current inputs and the exit-code/file-existence pass criteria. Then read comfy_cli/command/run/execution.py and ComfyUI's execution.py validation path. Before coding, confirm whether the project wants warning failures, expected-output comparisons, or comfyci integration; done means the selected assertion behavior is documented and enforced.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, python
- Domain
- ci-cd, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100