camunda / camunda/api-test-generator

Encode scenario templates in TBox/ABox (post-conditions, lifecycle assertions)

Open
#268 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
0
Forks
3
Avg merge
13h 41m
Merged PRs (30d)
23

Description

Problem

The semantic dependency graph answers which operations belong in a scenario and in what order — it's a data-flow object over semantic types. But it cannot express what should be true between or after those operations: post-conditions, observation steps, expected status codes. Those are temporal/modal properties, a different mathematical object.

Concretely, today's positive suite has a real test-quality gap: a DELETE scenario creates the resource (via the graph-derived prereq chain) and deletes it, but nothing asserts the resource is actually gone. The graph cannot fix this; assertions about post-states have to come from somewhere else.

Meanwhile, the ontology has accumulated several fields whose natural consumer would be such assertions, but which sit unused today:

  • observableVia on edges (#201, Lift 1)
  • producesStates / requiredStates on artifact rules (#252, Lifts 11/13)
  • revokedBy on edges (#224, just merged)

Proposal

Introduce scenario templates as a new TBox/ABox layer. A template is a declarative list of step kinds; the planner instantiates a template against an endpoint, filling graph-derivable steps from BFS and leaving template-prescribed steps as the source of post-condition assertions.

Architecture
Layer Answers Source
Semantic graph (existing ABox) Which operations, in what order Derived from spec + domain semantics
Scenario template (new TBox/ABox) What to assert between operations Hand-authored per config
Planner Composes the two into an EndpointScenario Code
Step kinds (initial sketch)
  • PREREQ-CHAIN(target) — graph fills this in (current BFS behaviour)
  • INVOKE(opId) — invoke a named operation (often referenced via an ontology field like revokedBy)
  • OBSERVE(opId) with optional expect: { status, present, absent } — query state and assert
Example templates

DELETE-lifecycle (consumes the graph + the deleted-resource's read op):

1. PREREQ-CHAIN(create R)        ← graph
2. INVOKE(target DELETE)         ← template
3. OBSERVE(GET R) expect status 404

Edge-lifecycle (consumes establishedBy + observableVia + revokedBy):

1. PREREQ-CHAIN(establish)       ← graph
2. OBSERVE(observableVia) expect present
3. INVOKE(revokedBy)
4. OBSERVE(observableVia) expect absent

State-transition (consumes producesStates + observableVia):

1. PREREQ-CHAIN(target)          ← graph
2. INVOKE(target) producing state S
3. OBSERVE(state observation op) expect S
Constraint: templates are NOT Turing-complete

Templates are declarative step lists with expected outcomes. All scheduling (BFS, oneOf variants, artifact selection, role binding) stays in the planner exactly as today. The template only adds which extra steps to interleave and what to assert — it does not replace the planner.

Symmetry with request-validation

The request-validation/ 400 suite is already template-driven (~24 malformed-request kinds × every op, expecting 400). A unified template framework would absorb both positive lifecycle templates and the existing 400 family under one TBox.

Phasing

This is a Lift-class change. Phased to land incrementally without disturbing the existing planner:

  1. Phase 1 — TBox + first template: ontology/vocabulary/scenario-template.schema.json defines the AST. configs/camunda-oca/ontology/scenario-templates.json ships one template (edge-lifecycle first — it has the most unused ontology backing and the smallest planner surface area).
  2. Phase 2 — planner instantiation: planner emits template-derived scenarios alongside existing BFS scenarios. Initially additive; existing scenarios untouched.
  3. Phase 3 — emitter: Playwright emitter renders OBSERVE steps as assertions (expect(response.status()).toBe(404) etc.).
  4. Phase 4 — migrate implicit shapes: lift the implicit resource-CRUD and state-transition shapes out of BFS into named templates. BFS reduces to the PREREQ-CHAIN filler. Eventually fold request-validation/ template iteration into the same framework.

Out of scope (initial issue)

  • Full temporal logic (LTL/CTL). Step sequences with expected outcomes are sufficient.
  • Replacing BFS. Templates compose with BFS; they don't supersede it.
  • Multi-config template inheritance. Each config ships its own ABox; share later if patterns emerge.

Discussion points

  • Should templates live alongside the existing per-config ABoxes (configs/<config>/ontology/) or in a shared ontology/templates/ directory?
  • How explicit should the expect vocabulary be? status: 404 vs higher-level absent / present predicates?
  • Should the L3 invariants assert template coverage (e.g. "every edge has an applicable edge-lifecycle template")?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with ontology/vocabulary/scenario-template.schema.json and configs/camunda-oca/ontology/scenario-templates.json, then inspect the planner and Playwright emitter entry points. Compare the existing request-validation/ template-driven 400 suite with the proposed step kinds and review how current BFS scenarios are emitted. A phased implementation is done when the schema, an initial config template, planner instantiation, and emitted OBSERVE assertions work without disturbing existing scenarios.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
api, backend-api-design, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.