picatz / picatz/flowstate

flowfile: the compiler checks every expression and never asks what it will cost — cel-go's static estimator, with `max_items`/`max_len` as size hints, can refuse at validate an expression that cannot fit `DefaultCostLimit` and report the bound in `flow audit`

Open
#1,504 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

cli dsl enhancement
Dominant language
Go
Stars
9
Forks
0
Avg merge
3h 3m
Merged PRs (30d)
509

Description

Observed behavior

checkExpressionTypes (pkg/flowstate/v1/flowfile/celcheck.go:175-242) produces a checked AST for every expression and discards it after reading the issues. Env.EstimateCost(ast, estimator) (cel-go v0.31.0, cel/env.go) computes a static [min, max] cost from exactly that AST, given a checker.CostEstimator that answers EstimateSize for a variable and EstimateCallCost for a function. Nothing in the tree calls it (grep EstimateCost at 40cc365: none).

The size hints are in the file. An input declares max_items: and max_len: (workflow.proto:973-1068, the buf.validate vocabulary), and checkExpressionInputTypes (flowfile/schema.go:202) already recognizes a bare ${inputs.x}; an inputs.hosts declared max_items: 100 bounds inputs.hosts.map(h, ...) at 100 iterations statically. Where nothing bounds a value (a step output) the estimate is unbounded, which is the honest answer and the one flow audit should print.

Today the budget is discovered at run time only. DefaultCostLimit is "a cost budget, not a time budget" (docs/reference/cel.md:113-118), so a comprehension over a 10k-item input fails an hour into a durable run rather than at flow validate. #1458 makes the same argument for optional inputs: the compiler knows and says nothing.

Desired outcome

Two projections of one estimate. flow audit (tier 0, measurement — flowfile/audit.go) reports per expression the static [min, max] cost against DefaultCostLimit, with "unbounded" where a dyn operand has no declared size. flow validate (tier 1) refuses an expression whose minimum estimate exceeds the budget, because that run cannot succeed on any input — the #1458 doctrine.

Acceptance criteria

  • flow audit on examples/matrix-fan-out/workflow.yaml prints an estimate for the items: comprehension bounded by the declared list sizes.
  • A fixture with max_items: 100000 on an input and a nested map over it is a positioned flow validate diagnostic naming the estimate and the budget; the same file with max_items: 10 validates.
  • The estimator's per-function costs are cel-go's own (ext/costs.go, at the library versions #1358 pins) plus byteCostEstimator's rules for json_parse and digest.sha256, so the static estimate and the runtime charge disagree only where sizes are unknown; a test compares them across the examples corpus.
  • The examples corpus produces zero new validate diagnostics.

Constraints and dependencies

  • The estimate reads the checked AST the compiler already has; with #1481 the same estimator becomes available to the runtime at plan time, but nothing here depends on it.
  • checker.CostEstimator sizes are in cel-go's units; byteCostEstimator (celcost.go) prices strings by size, and the static side must use the same factor (common.StringTraversalCostFactor) or the two budgets diverge.
  • #1360 (fusion) changes where charges land at run time; a static estimate over the expansion is unaffected.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with checkExpressionTypes in pkg/flowstate/v1/flowfile/celcheck.go and trace the checked AST into flowfile/audit.go and flow validate. Read cel-go's Env.EstimateCost, checker.CostEstimator, ext/costs.go, and the existing byteCostEstimator in celcost.go. Done means bounded per-expression audit output, positioned validation diagnostics for estimates over DefaultCostLimit, matching size and cost rules, and no new diagnostics in the examples corpus.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, compilers
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.