camunda / camunda/api-test-generator

Promote serverEmergent keys to runtimeEmission

Open
#305 1 comment 0 reactions 1 assignee View on GitHub

@jwulf is already working on this.

Since May 20, 2026.

OCA
Dominant language
TypeScript
Stars
0
Forks
3
Avg merge
13h 41m
Merged PRs (30d)
23

Description

Problem

The planner cannot generate a meaningful test for any mutation operation whose key is produced as a side-effect of process execution (updateUserTask, completeUserTask, assignUserTask, updateJob, failJob, resolveIncident, correlateMessage, …). It falls through to a synthetic seedBinding(...) placeholder, the PATCH/POST is dispatched against a key that doesn't exist, and the realistic outcome is 404/400 — not a real test of the endpoint.

Today this is hidden because every such operation is currently classified kind: "serverEmergent" in configs/camunda-oca/ontology/semantics.json with the comment "no direct create API → planner mints a deterministic placeholder". That classification is the surrender, not a description: the keys are reachable, the planner just isn't allowed to reach them.

A worked example: updateUserTask against the bundled spec emits

ctx.userTaskKeyVar = ctx.userTaskKeyVar ?? seedBinding('userTaskKeyVar');
const url = `${baseUrl}/user-tasks/${ctx.userTaskKeyVar || '${userTaskKey}'}`;
const resp1 = await request.patch(url, { headers: await authHeaders(), data: {} });
expect(resp1.status()).toBe(204);

— a synthetic key, an empty body, and a status assertion that cannot succeed on a real cluster.

Root cause (two intersecting defects)

1. Deployment escaped the prereq chain via the codegen role hook

configs/camunda-oca/codegen/playwright/roles/deploymentGateway/ deploys BPMN fixtures before the Playwright suite runs. That structure was an emitter ergonomic (so every test doesn't re-deploy the same BPMN), not a semantic replacement for declaring deployment as a planner-visible prerequisite. The planner therefore has no notion that createProcessInstance requires ModelDeployed, and no notion that some BPMN fixtures provide ModelHasUserTask / ModelHasServiceTask / etc.

The fix is to put deployment back into the prereq graph as a real producer step. The role hook can remain as an optimisation (e.g. dedupe-by-fixture-hash at emission time), but the planner has to see it.

2. runtimeEmission is not modelled

Several keys currently bucketed as serverEmergent are in fact discoverable: they appear on a known search endpoint, under eventual consistency, after a specific upstream side-effect (process execution, job activation, incident raising). They differ from true serverEmergent keys (e.g. internally-generated correlation IDs that no API surface returns) and they need a different planning strategy.

Proposed new semantic-type kind:

"UserTaskKey": {
  "kind": "runtimeEmission",
  "intersectsWith": "serverEmergent",
  "emittedBy": {
    "predecessor": "ProcessInstanceExists",
    "guardedBy": ["ModelHasUserTask"]
  },
  "discoveredVia": {
    "operationId": "searchUserTasks",
    "filterBy": "processInstanceKey",
    "extractKey": "userTaskKey",
    "consistency": "eventual"
  }
}

runtimeEmission intersects with serverEmergent — both describe keys with no direct create endpoint. The distinction is that runtimeEmission carries the planner information needed to reach the key: producing predecessor + discovery operation + consistency model. A pure-serverEmergent key (nothing discoverable) still falls through to placeholder, which is the correct outcome for that subset.

This needs:

  • A new edge kind in path-analyser/src/ontology/edgeSchema.ts (or a sibling vocabulary) for runtime emission — distinct from the existing membership edges (PUT/DELETE between two extant entity kinds). Membership edges describe what the API offers; runtime-emission edges describe what the system produces during execution.
  • Capability declarations in configs/camunda-oca/ontology/semantics.json (ModelHasUserTask, ModelHasServiceTask, ModelHasJob, ModelEmitsIncident, …) and corresponding fixture-metadata entries in configs/camunda-oca/fixtures/deployment-artifacts.json so the planner can pick the right BPMN for the capability the test needs.
  • Planner support in path-analyser/src/scenarioGenerator.ts: when a missing prereq is a runtimeEmission key, plan
    createDeployment(fixture-with-capability) → createProcessInstance(deployed) → poll search-with-eventual-consistency → bind key → operation under test.
  • The await-eventually helper at materializer/src/playwright/support/await-eventually.ts is already there; the planner just needs to wire it in.

Read-back after mutate — belongs in scenario-template vocabulary

The current path-analyser/src/ontology/scenarioTemplateSchema.ts encodes test patterns as PrereqChain / Invoke / Observe steps applied to Edge and Entity subjects. "Mutate then re-fetch and assert the field changed" is a fourth pattern with the same shape and belongs there, not in the planner's hard-coded chain logic. Sketch:

{
  "name": "UpdatedFieldVisibleOnReadBack",
  "appliesTo": { "kind": "RuntimeEntity" },     // or extend Entity
  "steps": [
    { "kind": "PrereqChain", "for": "mutator" },
    { "kind": "Invoke",      "op":  "mutator" },
    { "kind": "Observe",
      "op": "fetcher",
      "expect": "fieldEquals",
      "field": "<resolved-from-mutator-request-body>" }
  ]
}

This generalises beyond user tasks: every updateXxx / assignXxx / unassignXxx becomes a candidate for an automatic read-back-after-mutate scenario, without each one being baked into the planner.

Class scope

The defect applies to every serverEmergent-classified key that is in fact reachable via search:

  • UserTaskKeysearchUserTasks(processInstanceKey)
  • JobKeysearchJobs(processInstanceKey)
  • IncidentKeysearchIncidents(processInstanceKey)
  • MessageSubscriptionKeysearchMessageSubscriptions(processInstanceKey)
  • VariableKeysearchVariables(scopeKey)
  • DeploymentKey → returned synchronously from createDeployment (this one is closer to a missing producer wiring than a runtime-emission case, but it's in the same bucket today)
  • ProcessInstanceKey itself — already a producer, but a runtimeEmission view would make the dependency on ModelDeployed explicit instead of relying on the role hook

A targeted regression invariant: every operation that consumes a runtimeEmission key in a path-param either has a planner-derived discovery chain leading to that key, or the scenario is explicitly suppressed.

Acceptance criteria

  1. New ontology vocabulary: runtimeEmission semantic-type kind (intersecting with serverEmergent) with emittedBy predecessor + capability guards and discoveredVia search-op + consistency model.
  2. Capability + fixture metadata for BPMN fixtures (ModelHasUserTask first; others to follow).
  3. Deployment surfaced as a planner-visible producer of ModelDeployed + capabilities; the codegen deploymentGateway role degrades to an emission-time dedupe optimisation, not a semantic substitute.
  4. scenarioGenerator.ts plans the createDeployment → createProcessInstance → search-with-eventual-consistency → bind key → operation under test chain when the missing prereq is a runtimeEmission key.
  5. Observe step extended (or sibling step added) in scenario-template vocabulary to cover "read-back-after-mutate, assert field equals submitted value".
  6. A class-scoped L3 invariant under configs/camunda-oca/regression-invariants.test.ts: no operation that consumes a runtimeEmission key in a path-param compiles into a scenario whose only producer for that key is seedBinding(...) against a string placeholder.

Out of scope

  • Pure serverEmergent keys with no discovery surface (those continue to fall through to placeholders).
  • Negative tests for updateXxx on a non-existent key — separately useful, owned by request-validation.
  • Idempotency / cleanup of deployed BPMN between runs — separate concern, intersecting with #304 (uniqueness of test identifiers).

Notes

  • This is the root cause for the long tail of updateXxx / assignXxx / completeUserTask / etc. tests being effectively no-ops.
  • The fix is principled rather than mechanical: it gives the planner the same view the human author of an integration test has — "to update a thing, I need to make a thing, and to make a thing of this kind I need this BPMN to be deployed".

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.