galaxyproject / galaxyproject/foundry

summarize-cwl: tighten cast skill's deterministic/LLM split

Open
#231 0 comments 0 reactions 0 assignees View on GitHub
enhancement priority/v2 roadmap/substep
Dominant language
TypeScript
Stars
2
Forks
3
Avg merge
1d 22m
Merged PRs (30d)
63

Description

Follow-up from review of the `summarize-cwl` Mold + generated cast skill (branch `galaxy_cwl`). Two related concerns about the runtime contract the cast skill hands to the LLM.

## 1. Steps 3-6 are mechanical, not inferential

`SKILL.md` procedure delegates parse/normalize/validate to CLIs (`cwltool --validate`, `cwl-normalizer`, `validate-summary-cwl`) but asks the LLM to walk normalized cwl-utils JSON and emit `workflow_inputs`, `workflow_outputs`, `steps[]`, `tools[]`, and `graph` directly.

The Mold's research note argues against summarize-nextflow's heavy machinery on the grounds CWL doesn't need *inference*. True — but what's left in steps 3-6 is *extraction* over an already-typed object model, which is exactly where a small deterministic extractor (Python over `cwl-utils`, or TS) would be cheaper and safer than an LLM walker:

- schema validates shape, not coverage — an LLM can quietly drop steps on a large workflow and still pass `validate-summary-cwl`
- token cost scales linearly with workflow size for work that is mechanical JSON projection
- normalized cwl-utils output is regular JSON; there is no judgment call in steps 3-4

Revisit if eval cases (`content/molds/summarize-cwl/eval.md`) catch missing-step or wrong-tool-surface failures on real fixtures. If so, lift extraction into a `@galaxy-foundry/summarize-cwl` package paralleling summarize-nextflow, and leave the skill thin.

## 2. `graph.edges[].via` markers duplicate `steps[].in[]` fields

After this branch's schema changes, `steps[].in[]` already records `link_merge`, `pick_value`, `value_from`, and `default` per step input. The schema also asks `graph.edges[].via` to carry `linkMerge`, `pickValue`, `valueFrom`, `secondaryFiles`, and `scatter` markers.

For `linkMerge`/`pickValue`/`valueFrom` this is redundant — the LLM-built `via` array and the structured step-input fields can drift. Options:

- drop those three from `via` and keep it for `scatter` / `secondaryFiles` only
- or drop them from `steps[].in[]` and keep the graph as the single source

Either way, one place. Currently both, populated by the LLM independently.

## Scope

Both items live in the same surface (the LLM's contract in the cast bundle) and would be addressed by the same investigation: run the eval fixtures, see whether the LLM walker holds up, decide between thinning the skill or extracting a CLI. Not blocking the current branch.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.