galaxyproject / galaxyproject/foundry
summarize-cwl: tighten cast skill's deterministic/LLM split
- Dominant language
- TypeScript
- Stars
- 2
- Forks
- 3
- Avg merge
- 1d 22m
- Merged PRs (30d)
- 63
Description
Follow-up from review of the `summarize-cwl` Mold + generated cast skill (branch `galaxy_cwl`). Two related concerns about the runtime contract the cast skill hands to the LLM.
## 1. Steps 3-6 are mechanical, not inferential
`SKILL.md` procedure delegates parse/normalize/validate to CLIs (`cwltool --validate`, `cwl-normalizer`, `validate-summary-cwl`) but asks the LLM to walk normalized cwl-utils JSON and emit `workflow_inputs`, `workflow_outputs`, `steps[]`, `tools[]`, and `graph` directly.
The Mold's research note argues against summarize-nextflow's heavy machinery on the grounds CWL doesn't need *inference*. True — but what's left in steps 3-6 is *extraction* over an already-typed object model, which is exactly where a small deterministic extractor (Python over `cwl-utils`, or TS) would be cheaper and safer than an LLM walker:
- schema validates shape, not coverage — an LLM can quietly drop steps on a large workflow and still pass `validate-summary-cwl`
- token cost scales linearly with workflow size for work that is mechanical JSON projection
- normalized cwl-utils output is regular JSON; there is no judgment call in steps 3-4
Revisit if eval cases (`content/molds/summarize-cwl/eval.md`) catch missing-step or wrong-tool-surface failures on real fixtures. If so, lift extraction into a `@galaxy-foundry/summarize-cwl` package paralleling summarize-nextflow, and leave the skill thin.
## 2. `graph.edges[].via` markers duplicate `steps[].in[]` fields
After this branch's schema changes, `steps[].in[]` already records `link_merge`, `pick_value`, `value_from`, and `default` per step input. The schema also asks `graph.edges[].via` to carry `linkMerge`, `pickValue`, `valueFrom`, `secondaryFiles`, and `scatter` markers.
For `linkMerge`/`pickValue`/`valueFrom` this is redundant — the LLM-built `via` array and the structured step-input fields can drift. Options:
- drop those three from `via` and keep it for `scatter` / `secondaryFiles` only
- or drop them from `steps[].in[]` and keep the graph as the single source
Either way, one place. Currently both, populated by the LLM independently.
## Scope
Both items live in the same surface (the LLM's contract in the cast bundle) and would be addressed by the same investigation: run the eval fixtures, see whether the LLM walker holds up, decide between thinning the skill or extracting a CLI. Not blocking the current branch.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.