camunda / camunda/api-test-generator
Encode scenario templates in TBox/ABox (post-conditions, lifecycle assertions)
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 3
- Avg merge
- 13h 41m
- Merged PRs (30d)
- 23
Description
Problem
The semantic dependency graph answers which operations belong in a scenario and in what order — it's a data-flow object over semantic types. But it cannot express what should be true between or after those operations: post-conditions, observation steps, expected status codes. Those are temporal/modal properties, a different mathematical object.
Concretely, today's positive suite has a real test-quality gap: a DELETE scenario creates the resource (via the graph-derived prereq chain) and deletes it, but nothing asserts the resource is actually gone. The graph cannot fix this; assertions about post-states have to come from somewhere else.
Meanwhile, the ontology has accumulated several fields whose natural consumer would be such assertions, but which sit unused today:
observableViaon edges (#201, Lift 1)producesStates/requiredStateson artifact rules (#252, Lifts 11/13)revokedByon edges (#224, just merged)
Proposal
Introduce scenario templates as a new TBox/ABox layer. A template is a declarative list of step kinds; the planner instantiates a template against an endpoint, filling graph-derivable steps from BFS and leaving template-prescribed steps as the source of post-condition assertions.
Architecture
| Layer | Answers | Source |
|---|---|---|
| Semantic graph (existing ABox) | Which operations, in what order | Derived from spec + domain semantics |
| Scenario template (new TBox/ABox) | What to assert between operations | Hand-authored per config |
| Planner | Composes the two into an EndpointScenario |
Code |
Step kinds (initial sketch)
PREREQ-CHAIN(target)— graph fills this in (current BFS behaviour)INVOKE(opId)— invoke a named operation (often referenced via an ontology field likerevokedBy)OBSERVE(opId)with optionalexpect: { status, present, absent }— query state and assert
Example templates
DELETE-lifecycle (consumes the graph + the deleted-resource's read op):
1. PREREQ-CHAIN(create R) ← graph
2. INVOKE(target DELETE) ← template
3. OBSERVE(GET R) expect status 404
Edge-lifecycle (consumes establishedBy + observableVia + revokedBy):
1. PREREQ-CHAIN(establish) ← graph
2. OBSERVE(observableVia) expect present
3. INVOKE(revokedBy)
4. OBSERVE(observableVia) expect absent
State-transition (consumes producesStates + observableVia):
1. PREREQ-CHAIN(target) ← graph
2. INVOKE(target) producing state S
3. OBSERVE(state observation op) expect S
Constraint: templates are NOT Turing-complete
Templates are declarative step lists with expected outcomes. All scheduling (BFS, oneOf variants, artifact selection, role binding) stays in the planner exactly as today. The template only adds which extra steps to interleave and what to assert — it does not replace the planner.
Symmetry with request-validation
The request-validation/ 400 suite is already template-driven (~24 malformed-request kinds × every op, expecting 400). A unified template framework would absorb both positive lifecycle templates and the existing 400 family under one TBox.
Phasing
This is a Lift-class change. Phased to land incrementally without disturbing the existing planner:
- Phase 1 — TBox + first template:
ontology/vocabulary/scenario-template.schema.jsondefines the AST.configs/camunda-oca/ontology/scenario-templates.jsonships one template (edge-lifecycle first — it has the most unused ontology backing and the smallest planner surface area). - Phase 2 — planner instantiation: planner emits template-derived scenarios alongside existing BFS scenarios. Initially additive; existing scenarios untouched.
- Phase 3 — emitter: Playwright emitter renders
OBSERVEsteps as assertions (expect(response.status()).toBe(404)etc.). - Phase 4 — migrate implicit shapes: lift the implicit
resource-CRUDandstate-transitionshapes out of BFS into named templates. BFS reduces to thePREREQ-CHAINfiller. Eventually foldrequest-validation/template iteration into the same framework.
Out of scope (initial issue)
- Full temporal logic (LTL/CTL). Step sequences with expected outcomes are sufficient.
- Replacing BFS. Templates compose with BFS; they don't supersede it.
- Multi-config template inheritance. Each config ships its own ABox; share later if patterns emerge.
Discussion points
- Should templates live alongside the existing per-config ABoxes (
configs/<config>/ontology/) or in a sharedontology/templates/directory? - How explicit should the
expectvocabulary be?status: 404vs higher-levelabsent/presentpredicates? - Should the L3 invariants assert template coverage (e.g. "every edge has an applicable edge-lifecycle template")?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ontology/vocabulary/scenario-template.schema.json and configs/camunda-oca/ontology/scenario-templates.json, then inspect the planner and Playwright emitter entry points. Compare the existing request-validation/ template-driven 400 suite with the proposed step kinds and review how current BFS scenarios are emitted. A phased implementation is done when the schema, an initial config template, planner instantiation, and emitted OBSERVE assertions work without disturbing existing scenarios.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- api, backend-api-design, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100