Comfy-Org / Comfy-Org/ComfyUI_frontend

Automate Core custom-node pin freshness and separate exclusion warnings

Open
#16,153 0 comments 0 reactions 0 assignees View on GitHub
area:testing Custom Node custom-nodes-canary-core Public API
Dominant language
TypeScript
Stars
2k
Forks
699
Avg merge
1d 7h
Merged PRs (30d)
490

Description

Main's nightly failed against a stale VHS pin; `VHS_SelectLatest` was an intentional coverage warning, not the failing assertion.

- VHS already fixed the real Nodes 2.0 crash in [upstream PR #709](https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite/pull/709).
- Frontend PR [#15590](https://github.com/Comfy-Org/ComfyUI_frontend/pull/15590) updated the Core pin and expectation matrices; a later main run is [green](https://github.com/Comfy-Org/ComfyUI_frontend/actions/runs/33133462893).
- Follow-up: make Core pin drift fail early and separately from intentional node-exclusion reporting.

Full context for agent readers

## Evidence

Nightly [run 33073597130](https://github.com/Comfy-Org/ComfyUI_frontend/actions/runs/33073597130) tested frontend `017660559f20` and failed on cloud shards 2/5 and 4/5 plus Core. The Core job's actual failures were:

- S2: VHS's `fitText` received a numeric annotated-widget label and called `text.slice`, producing `text.slice is not a function` while mounting all nodes under Vue Nodes.
- S3: four VHS upload widget entries in `ROUNDTRIP_VALUE_ALLOWED_KEYS` had become stale.
- S9: `VHS_SelectLatest` was explicitly excluded, then the remaining VHS corpus passed: 31 nodes auto-ran clean and the pack reported `result=pass`. See the [Core job](https://github.com/Comfy-Org/ComfyUI_frontend/actions/runs/33073597130/job/98521906529).

The tested frontend manifest pinned VHS to [`4ee72c065d`](https://github.com/Comfy-Org/ComfyUI_frontend/blob/017660559f20e7b76cdf07209fb7197ec1b8ad3d/browser_tests/fixtures/data/customNodeManifest.core.json#L27-L39), dated 2026-08-14. VHS had already fixed the S2 failure by coercing labels through `String(text)` in [PR #709](https://github.com/Kosinkadink/ComfyUI-VideoHelperSuite/pull/709), merged as `115de7a9d9` on 2026-08-26.

Frontend PR [#15590](https://github.com/Comfy-Org/ComfyUI_frontend/pull/15590) subsequently pinned VHS to `115de7a9d9`, recalibrated the round-trip matrix, and merged on 2026-08-27. Main [run 33133462893](https://github.com/Comfy-Org/ComfyUI_frontend/actions/runs/33133462893) is green at the resulting configuration.

## Root cause

This was custom-node source drift plus a stale frontend test ledger, not a frontend prompt-conversion/node-replacement regression. The sentence about `VHS_SelectLatest` came from the suite's intentional [S9 exclusion](https://github.com/Comfy-Org/ComfyUI_frontend/blob/017660559f20e7b76cdf07209fb7197ec1b8ad3d/browser_tests/fixtures/customNode/autoRun.ts#L189-L200); it was emitted as an annotation beside unrelated failing tiers and was mistaken for the failure.

## Proposed follow-up

1. Add a deterministic Core-pin freshness/synchronization check that compares duplicated Core and census pins before installing packs and reports the exact stale pack.
2. Add a unit repro for stale Core pin detection and retain the existing S2/S3 E2E coverage that caught this incident.
3. Add a matrix poison case where the Core pin is deliberately older while `VHS_SelectLatest` remains intentionally excluded. The verdict must fail for pin drift and must not attribute the failure to the exclusion warning.
4. Keep pack-owned compatibility fixes upstream. VHS PR #709 is the precedent; frontend should own pin provenance, expectation synchronization, and diagnostic attribution.
5. Document the extension-contract lesson: a custom-node pin, its expectation ledger, and its exclusion ledger form one reviewed compatibility snapshot and must move together.

Contributor guide

Open the contributing guide

Research direction

Start with browser_tests/fixtures/customNode/autoRun.ts and the Core manifest referenced in browser_tests/fixtures/data/customNodeManifest.core.json, then trace the existing S2/S3 E2E and pack-install flow. Add the proposed stale-pin unit and matrix coverage, keeping VHS_SelectLatest excluded without masking pin drift. Done means duplicated Core and census pins fail deterministically with the stale pack identified and the exclusion warning separately reported.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, typescript
Domain
ci-cd, testing
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
46/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.