flowfile: the compiler checks every expression and never asks what it will cost — cel-go's static estimator, with `max_items`/`max_len` as size hints, can refuse at validate an expression that cannot fit `DefaultCostLimit` and report the bound in `flow audit`
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 9
- Forks
- 0
- Avg merge
- 3h 3m
- Merged PRs (30d)
- 509
Description
Observed behavior
checkExpressionTypes (pkg/flowstate/v1/flowfile/celcheck.go:175-242) produces a checked AST for every expression and discards it after reading the issues. Env.EstimateCost(ast, estimator) (cel-go v0.31.0, cel/env.go) computes a static [min, max] cost from exactly that AST, given a checker.CostEstimator that answers EstimateSize for a variable and EstimateCallCost for a function. Nothing in the tree calls it (grep EstimateCost at 40cc365: none).
The size hints are in the file. An input declares max_items: and max_len: (workflow.proto:973-1068, the buf.validate vocabulary), and checkExpressionInputTypes (flowfile/schema.go:202) already recognizes a bare ${inputs.x}; an inputs.hosts declared max_items: 100 bounds inputs.hosts.map(h, ...) at 100 iterations statically. Where nothing bounds a value (a step output) the estimate is unbounded, which is the honest answer and the one flow audit should print.
Today the budget is discovered at run time only. DefaultCostLimit is "a cost budget, not a time budget" (docs/reference/cel.md:113-118), so a comprehension over a 10k-item input fails an hour into a durable run rather than at flow validate. #1458 makes the same argument for optional inputs: the compiler knows and says nothing.
Desired outcome
Two projections of one estimate. flow audit (tier 0, measurement — flowfile/audit.go) reports per expression the static [min, max] cost against DefaultCostLimit, with "unbounded" where a dyn operand has no declared size. flow validate (tier 1) refuses an expression whose minimum estimate exceeds the budget, because that run cannot succeed on any input — the #1458 doctrine.
Acceptance criteria
flow auditonexamples/matrix-fan-out/workflow.yamlprints an estimate for theitems:comprehension bounded by the declared list sizes.- A fixture with
max_items: 100000on an input and a nestedmapover it is a positionedflow validatediagnostic naming the estimate and the budget; the same file withmax_items: 10validates. - The estimator's per-function costs are cel-go's own (
ext/costs.go, at the library versions #1358 pins) plusbyteCostEstimator's rules forjson_parseanddigest.sha256, so the static estimate and the runtime charge disagree only where sizes are unknown; a test compares them across the examples corpus. - The examples corpus produces zero new validate diagnostics.
Constraints and dependencies
- The estimate reads the checked AST the compiler already has; with #1481 the same estimator becomes available to the runtime at plan time, but nothing here depends on it.
checker.CostEstimatorsizes are in cel-go's units;byteCostEstimator(celcost.go) prices strings by size, and the static side must use the same factor (common.StringTraversalCostFactor) or the two budgets diverge.- #1360 (fusion) changes where charges land at run time; a static estimate over the expansion is unaffected.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with checkExpressionTypes in pkg/flowstate/v1/flowfile/celcheck.go and trace the checked AST into flowfile/audit.go and flow validate. Read cel-go's Env.EstimateCost, checker.CostEstimator, ext/costs.go, and the existing byteCostEstimator in celcost.go. Done means bounded per-expression audit output, positioned validation diagnostics for estimates over DefaultCostLimit, matching size and cost rules, and no new diagnostics in the examples corpus.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, compilers
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 35/100