control-toolbox / control-toolbox/CTParser.jl

[Roadmap] Consolidate CTParser — separate parsing from semantics via an IR

Open
#289 1 comment 0 reactions 1 assignee Claimed by @jbcaillau View on GitHub
roadmap
Dominant language
Julia
Stars
3
Forks
0
Avg merge
4h 22m
Merged PRs (30d)
2

Description

## Objective

Refactor CTParser following the [control-toolbox Handbook philosophy](https://github.com/control-toolbox/Handbook) to build a better base for future extensions.

## Context

CTParser (v0.8.15) turns the `@def` DSL into executable model-building code, through two parsing backends living side by side in `onepass.jl` (~1500 lines): `:fun` generates function-based CTModels code, `:exa` generates a full ExaModels transcription (variables on a grid, scheme-dependent dynamics defects, cost quadrature, constraints). The package is a flat module with generated code addressed through mutable prefix `Ref`s. A [detailed roadmap](https://github.com/control-toolbox/CTParser.jl/discussions/288) has been written, companion to the [CTDirect roadmap](https://github.com/control-toolbox/CTDirect.jl/discussions/609).

## Guiding principle

**The parser describes the problem; it does not discretize it.** Scheme-dependent code (defects, quadrature, endpoint-control fixes, grids) belongs to CTDirect.

## Key points (from the [detailed roadmap](https://github.com/control-toolbox/CTParser.jl/discussions/288))

1. **ExaModels 0.11 migration** (§1) — Compat blocker for the whole ecosystem (CTDirect and CTSolvers also pin ExaModels = "0.9"). Audit all generated ExaModels call sites, bump compat, coordinate the bump across packages.
2. **Move the discretization out of the parser** (§2, the structural item) — The `:exa` backend stops generating scheme-dependent transcription code. Two routes: (1) **function tracing** — CTDirect traces CTModels functions into ExaModels generators, CTParser needs no `:exa` backend at all; (2) **expression IR** — CTParser stores a neutral expression representation, CTDirect generates the Exa transcription from it. Joint with CTDirect §5. Absorbs issues #65 (non-uniform grid), #161 (endpoint-control constraints), #134 (exa multipliers).
3. **"1-D = scalar" simplification** (§3) — Emitted closures should work when callers pass scalars for 1-D state/control/variable. Coordinate with CTDirect §7 and CTModels.
4. **Remove prefixes** (§4, #224) — Replace mutable prefix `Ref`s (`PREFIX_FUN`, `PREFIX_EXA`) with direct module-qualified code (`CTModels.…`, `CTBase.…`). Removes global mutable state, makes generated code readable and greppable.
5. **Parsing robustness** (§5) — Bug sweep (#197: match-pattern typos, `as_range` issues, `__wrap` scoping), stricter parsing with `ParsingError` for silent mis-parses (#193: leftover symbols, #139: undefined time arguments), finalize slice notation (#260: `x[1:3](t)`, `x[1:end](t)`, `x[1:n](t)`).
6. **Syntax extensions** (§6) — Coordinate-derivative aliases (`ẋ₁(t)` as sugar for `∂(x₁)(t)`, #71), tensors/matrix-valued state and control (#181, cross-package design note first), incremental `@def!` with `$`-interpolation (#32).
7. **Parsing engine redesign** (§7) — Staged compiler-style pipeline over a typed IR: Stage 0 (surface normalization), Stage 1 (grammar → typed IR nodes), Stage 2 (semantic validation: symbol table, constraint classification, dimension resolution, strictness checks), Stage 3 (backend emission). Replaces the 1500-line monolithic `onepass.jl`. The IR is route 2's neutral representation — engine redesign and Exa repatriation converge on the same object. Submodule split follows the stages (IR, Grammar, Semantics, FunBackend, ExaBackend, InitialGuess, Defaults).
8. **Test suite restructuring** (§8) — Adopt the Handbook testing template, organize by functionality (grammar, strictness, codegen_fun, codegen_exa, initial_guess, integration), keep `:exa` tests as golden oracle for CTDirect's repatriation.
9. **Documentation** (§9) — Vitepress migration + a complete DSL grammar reference page derived from the §7 IR productions, with doctests.

## Why

Separating parsing from semantics via an IR will make the DSL more maintainable and extensible:
- New transformations (multi-phase, penalization, symbolic reduction) can operate at the IR level without touching the parser
- Adding a production = adding one file (node struct + parse rule + validate method + emit methods) — plain open multiple dispatch
- Adding a backend = adding emit methods, not duplicating the grammar
- Removing the ExaModels transcription clarifies the responsibility boundary with CTDirect
- Strictness rules become centralized stage-2 checks on data instead of guards scattered through codegen
- Error accumulation: one `@def` can report all its errors with positions in one shot

## References

- [CTParser roadmap discussion](https://github.com/control-toolbox/CTParser.jl/discussions/288)
- [CTDirect roadmap discussion](https://github.com/control-toolbox/CTDirect.jl/discussions/609) (companion)
- [Handbook](https://github.com/control-toolbox/Handbook)
- [Roadmap comment](https://github.com/orgs/control-toolbox/discussions/68#discussioncomment-17584843)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.