Request-validation suite (camunda-oca, unsecured): 280 failures across 11 root causes

The camunda-oca request-validation suite (unsecured profile) reports 280 failures out of 1906
tests, 14.7%. None of them are flakes. Every one is a status mismatch, and the run records 0 flaky
and 0 skipped. Dated 2026-08-31, from
generated/camunda-oca/request-validation/unsecured/playwright-report.

I classified all 280 into 11 disjoint root causes and checked each against
spec/camunda-oca/bundled/rest-api.bundle.json and a live server. Roughly two thirds come from
defects in the generation chain. The rest are real conformance gaps where the generated test is right
and the API is wrong. Those two halves need opposite treatment. Fix the generator for the first, file
upstream for the second.

This is the parent issue. Each item in the checklists below gets its own sub-issue linking back here,
and every knownIssue.tracker field added to configs/camunda-oca/request-validation.json should
point at this one.

Read this first: the baseline is not reproducible

The container that produced the report no longer exists. It ran from docker/docker-compose.yml with
CAMUNDA_DATA_SECONDARY_STORAGE_TYPE: rdbms, CAMUNDA_SECURITY_AUTHORIZATIONS_ENABLED: false and
UNPROTECTEDAPI: true, on image tag 8.9-SNAPSHOT that self-reported 8.10.0-alpha4. Something
replaced it mid-investigation with camunda/camunda:SNAPSHOT on Elasticsearch, requiring auth. The
two disagree on at least 47 results:

Probe	rdbms, unprotected	Elasticsearch, authenticated
POST /v2/agent-definitions/search	404	200
POST /v2/authentication/me/authorizations/search	404	200
POST /v2/secrets/list	404	200
GET /v2/backups/history	404	403
{"page":{"limit":10001}}	200	500

Groups A, F and G reproduce identically on both, so those conclusions stand. Everything else needs
re-measuring against a clean baseline before anyone changes code. Hence the CI workflows and the
re-baseline coming first.

Scope is the unsecured profile only. Nobody has ever run secured/ or rbac/. Neither has a
test-results.json. The group A fix alone changes 22 emitted tests in secured/, so this work
touches code with no baseline behind it. A green unsecured result does not mean all three profiles
are covered.

Failure taxonomy

#	Root cause	Count	Side	Confidence
A	Path-item servers: override ignored, so /v2 gets double-prefixed	16	generator	High
B	oneOf pagination maximum treated as unconditional	82	generator	High
C	Seeded path-param key 1 resolves to a missing resource	34	generator	High
D	Routes unregistered on the report's deployment	47	environment	High
E	Tests emitted for one-shot precondition operations	12	generator	High
F	Path-param pattern and maxLength not enforced	24	API	High
G	page.limit: 0 accepted, spec says minimum: 1	41	API	High
H	operationReference minimum not enforced	12	API	High
I	Blank and whitespace tenantId accepted	8	API	High
J	500 on backup operations under rdbms secondary storage	2	API	Medium
K	Undecodable cursor accepted by one endpoint	2	API	Medium

---

Group A: path-item servers: override ignored (16)

Symptom. PATCH /v2/cluster/v2/mode and POST /v2/cluster/v2/restore return
"No static resource v2/cluster/v2/mode.".

Root cause. The 14 cluster-admin path items each override servers: with a URL that drops the
document's /v2 suffix, because /cluster/v2 already lives in the path key.
cluster-admin.yaml:1-12 spells this out. The generator reads servers nowhere at all.
request-validation/src/spec/loader.ts:223 stores the raw path key, and
request-validation/templates/support/http.ts:42 prepends /v2 to everything.

Evidence. High confidence, reproduced on both deployments. GET /cluster/v2/status returns 204 on
rdbms and 200 on Elasticsearch. GET /v2/cluster/v2/status returns 404 and 401. The bundle confirms
paths['/cluster/v2/mode'].servers[0].url lacks the /v2 the root server carries.

Fix, part one. Carry the effective base per operation from loader.ts into the scenario, and give
buildUrl an optional base argument instead of the module constant API_VERSION. This also corrects
22 tests in the secured/ suite.

Fix, part two: cluster-admin credentials. Correcting the URL alone only moves these from 404 to
401. Cluster-admin operations sit behind their own credential chain
(ClusterAdminBasicSecurityConfiguration, cluster-admin.yaml:26-31), and an Orchestration Cluster
session does not authenticate against it. The suite already has a precedent for a second credential
set: denyProbeCredentials at templates/support/env.ts:41-45, exported by
scripts/e2e/run-oca.sh:58-64. Follow that shape.

1. Add clusterAdminCredentials and a clusterAdminHeaders() helper to env.ts.
2. Mark cluster-admin-server operations in loader.ts, carry the flag on the scenario, and have
   qaEmitter.ts emit clusterAdminHeaders() for them instead of jsonHeaders().
3. Export the env pair from run_rv() in scripts/e2e/run-oca.sh.

Find the credentials before writing any of that. No compose file sets a cluster-admin credential,
so nobody knows what they are here. Check the container startup log for a generated password, or set
an explicit pair in compose. Confirm with curl -u <user>:<pass> http://localhost:8080/cluster/v2/topology returning 200. If no workable credential exists, fall back
to excludeOperations for the two operations and ship the servers: fix anyway, since it corrects
the secured suite either way.

Group B: oneOf pagination maximum treated as unconditional (82)

Symptom. {"page":{"limit":10001}} and {"page":{"limit":10100}} expect 400. All 94 emitted
tests of this shape fail, everywhere in the suite. Not one passes.

Root cause. SearchQueryPageRequest is a oneOf over four branches.
request-validation/src/analysis/paginationLimit.ts:23-32 picks the branch whose only property is
limit, which is LimitPagination with minimum: 1 and maximum: 10000, then derives mutations
from it at :34-45 without looking at the siblings. OffsetPagination.limit declares minimum: 1
and no maximum. So {limit: 10001} still satisfies a branch, the payload is contract-valid, and
the generator is asserting a constraint the schema never imposed.

Evidence. High confidence. Schema at search-models.yaml:37-51. Reproduced live: 10001 and 10100
return 200 on rdbms and 500 on Elasticsearch, never 400, while -99 returns 400 on both. The
suite-wide pass rate lines up exactly. wayBelowMinimum (-99) passes 45 of 47. aboveMaximum and
wayAboveMaximum pass 0 of 47 each.

Fix. In planLimitMutations, keep a mutation only when it violates every branch of the oneOf,
not just the limit-only branch. paginationShape.ts already hands over page.branches. That drops
both above-maximum variants, keeps both below-minimum ones, and generalises to any future oneOf
envelope.

Group C: seeded path-param key resolves to a missing resource (34)

Symptom. POST /v2/element-instances/1/incidents/search with a malformed body returns
{"detail":"Element Instance with key '1' not found"} rather than the body-validation 400.

Root cause. Body-scenario generators fill path params with the literal '1'
(paginationLimit.ts:83-89 and siblings). The server resolves the path resource before it validates
the body, so the test never reaches the layer it was written to exercise.

Affected. searchElementInstanceIncidents, searchProcessInstanceIncidents,
searchUserTaskAuditLogs and searchUserTaskVariables at 6 each, plus deleteProcessInstance,
deleteDecisionInstance, resolveIncident, updateJob and correlateMessage at 2 each.
correlateMessage is the odd one out. Its 404 comes from a body-level lookup, "no subscription for
message with name X", not a path param, so it needs a running instance with a message subscription
rather than a key.

Evidence. High confidence. The server's own 404 detail strings name the unresolved resource.

Fix, with a mechanism that already exists and nobody wired up here. resourceFixtures and
pathResourceFixtures (request-validation/src/config.ts:108-118, #352) rewrite the filler to
process.env['RV_FIXTURE_*'] || '1', so a malformed-field test rides on a real resource.
configs/camunda-hub/request-validation.json:9-19 is the worked example.
configs/camunda-oca/request-validation.json uses none of it.

Add the fixture map plus a standalone make_fixtures step in scripts/e2e/run-oca.sh, mirroring
run-hub.sh. Standalone rather than harvesting keys from the positive suite, so the negative suite
keeps working under SKIP_POSITIVE=1. Every BPMN artifact needed already sits in
configs/camunda-oca/fixtures/bpmn/: simple.bpmn for a process and element instance,
user-task.bpmn, service-task.bpmn for a job, incident-script-task.bpmn for an incident, and
message-catch-event.bpmn or service-message-catch.bpmn for a message subscription.

Group D: routes unregistered on the report's deployment (47)

Symptom. No static resource ... on /v2/authentication/me/authorizations/search (15),
/v2/agent-definitions/* (14), /v2/backups/history* (15) and /v2/secrets/list (3).

Two hypotheses I disproved. First, spec-ahead-of-server skew. All four carry
x-added-in-version: "8.10", but the server self-reported 8.10.0-alpha4, and searchAgentInstances
is also marked 8.10 and returned 200. Second, absent from the build. On the Elasticsearch container,
authenticated, all four routes respond, three with 200 and backups with 403. The routes exist.

Root cause. Deployment configuration. Not the generator, not the suite. I did not isolate which
setting is responsible. CAMUNDA_DATA_SECONDARY_STORAGE_TYPE: rdbms would explain history backups,
which need Elasticsearch or OpenSearch, and CAMUNDA_SECURITY_AUTHORIZATIONS_ENABLED: false would
explain me/authorizations. Neither explains agent definitions or secrets. The image tag differs too,
8.9-SNAPSHOT against SNAPSHOT.

Evidence. High confidence that the routes exist and the cause is environmental. Low confidence on
which setting does it.

Fix: correct the deployment, do not exclude the tests. An exclusion in
configs/camunda-oca/request-validation.json applies to the whole config, since no deployment
dimension exists. It would delete searchAgentDefinitions, searchOwnAuthorizations, listSecrets
and the history-backup operations from every run, including the ones where they work fine.

Treat the compose correction as the first experiment of the re-baseline, and watch for a side effect.
The unsecured profile depends on UNPROTECTEDAPI: true with authorizations off, which is how requests
reach the API with no credentials at all. Every unauthenticated call on the Elasticsearch container
returned 401. Turning authorizations on may change what "unsecured" means for the whole suite, so
measure before committing to it. If the corrected deployment trades 47 failures for a larger
auth-shaped set, then the team is genuinely running two deployment shapes, and the honest answer is a
deployment dimension in the config layer: a scoped block in request-validation.json, a validator
beside config.ts:227-249, selection via an env var alongside RV_PROFILE, threading through
generate.ts:193-204, and guardrail tests.

Group E: one-shot precondition operations (12)

Symptom. Nine createInitialAdminUser tests return 403, "Expected to create an initial admin user, but found existing admin users". Three restoreAsClusterAdmin tests return 409, "Restore is only allowed while the cluster is in recovery mode".

Root cause. Both check a global precondition before validating anything. A fresh container does
not rescue them. playwright.config.ts sets fullyParallel: true, so on a pristine server exactly
one of the nine would win the race and the other eight would still get 403. These two operations are
structurally untestable by a validation suite. Transient state is not the problem.

Evidence. High confidence. The server states the reason in the response body, and the parallel
race follows from the config.

Fix. excludeOperations entries for createInitialAdminUser and restoreAsClusterAdmin, each
with a knownIssue block.

---

API-side gaps (89): file upstream, then reference from knownIssues

These are the suite working as intended, catching places where the server does not enforce what the
spec declares. knownIssues is reporting metadata. It does not drop tests, so these stay red until
the API changes. I think that is the right outcome: visible in the report, labelled with a tracking
issue, rather than buried in an exclusion list. It also means the nightly is permanently red by
design.

F. Path-param constraints not enforced (24). High confidence. RoleId, TenantId, Username and
MappingRuleId declare a pattern. GroupId declares maxLength: 256. The server enforces none of
them on path params. Reproduced on both deployments: POST /v2/roles/!/users/search returns 200 with
an empty page, GET /v2/roles/! returns 404, and GET /v2/roles/<266 a's> returns 404. Never 400.
The two response shapes differ only in what the handler does downstream, so they are one cause.

Expect pushback here. A reviewer can reasonably argue 404 is fine for an unmatchable path segment. If
upstream declines, the fallback is an excludeOperations entry scoped to
param-constraint-violation, with the reasoning written into reason.

G. page.limit: 0 accepted (41). High confidence. All four oneOf branches declare minimum: 1,
so 0 violates every one of them. Reproduced on both deployments: {"limit":0} returns 200 and
{"limit":-99} returns 400. The server looks like it only checks limit < 0.

H. operationReference minimum not enforced (12). High confidence. The schema declares
"type": "integer", "format": "int64", "minimum": 1, and the description says "Must be > 0 if
provided." The server takes 0 and -99 on all six process-instance batch operations.

I. Blank and whitespace tenantId accepted (8). High confidence. publishMessage,
broadcastSignal and getProcessDefinitionInstanceVersionStatistics all accept "" and "\n".
Both fields resolve through allOf to TenantId, which declares minLength: 1 and
pattern: ^(<default>|[\w\.\-]{1,31})$.

J. 500 on backup operations (2). Medium confidence. "Unexpected error occurred between gateway and broker (code: UNSUPPORTED_MESSAGE)". This looks like the same rdbms limitation as group D,
except it returns a 500 instead of a 404. Re-check after the re-baseline. If it vanishes on
Elasticsearch it belongs with group D. If it survives, a 500 for an unsupported configuration is worth
filing on its own.

K. Cursor not validated (2). Medium confidence. searchProcessDefinitionVariableNames accepts an
undecodable cursor and returns 200, while 40 other endpoints return 400 for the identical payload.

---

A separate generator defect, found while tracing group F

request-validation/scripts/generate.ts:609-623 builds its dedupe key from
method | path | type | target | bodyEncoding | bodyHash and leaves out constraintKind. For
param-constraint-violation scenarios there is no body, and target is just path.tenantId, so
every violation kind for one parameter collapses into a single scenario and only the first survives.

I checked all 126 path params in the bundle. Every one declares both a pattern and a
maxLength, so every one loses its length test today. groupId keeps its length test only because
it has no pattern.

Adding constraintKind to the key restores about 126 tests. Around 72 of them, on string identifiers,
will fail because of group F. The other 54 sit on numeric keys, where 266 characters of a also
violates the numeric pattern, so those should return 400 and pass, exactly as their existing pattern
tests already do. 92 of the path pattern tests pass today. Ship it anyway and fold the 72 into the
group F issue as more evidence.

You cannot suppress only the length cases. excludeOperations[].scenarioKinds matches s.type, and
pattern and length-max share the type param-constraint-violation. constraintKind does exist on the
scenario at model/types.ts:174, but nothing filters on it.

---

Sub-issues

In recommended merge order. Sub-issue N produces PR N.

• 0. CI workflows. No OCA nightly exists. ci.yml touches camunda-oca only for spec-pin
  freshness (:338-359) and regression invariants, and every nightly and triage workflow is Hub-only.
  That is why a human had to find these 280 by hand, and why the next regression would go unnoticed.
  Add three files. _oca-suite-run.yml is the reusable one: start the container from
  docker/docker-compose.yml, run RV_PROFILES=unsecured scripts/e2e/run-oca.sh, write the
  summarize-failures.mjs breakdown to $GITHUB_STEP_SUMMARY, and upload test-results.json,
  junit-report.xml and playwright-report/. Then nightly-camunda-oca.yml on cron plus
  workflow_dispatch, and oca-ondemand-test.yml with profile and steps inputs, modelled on the
  47-line hub-ondemand-test.yml. All far smaller than the Hub equivalents, which need Keycloak,
  Vault, OIDC and a Slack app. Gating is strict, matching the Hub nightly, which means this
  nightly is red from its first run. Read the count, not the badge.
• 0b. Re-baseline. Run the new on-demand workflow and record the result. Try the group D
  compose correction here as the first experiment, and measure what enabling authorizations does to
  the unsecured profile before committing. Everything below gets measured against this run.
• 1. oneOf-aware limit mutations. analysis/paginationLimit.ts, analysis/paginationShape.ts.
  Removes 94 emitted tests and clears 82 failures. Shares no files with anything else.
• 5. Dedupe key. One line in generate.ts:609-623. Adds about 126 tests and raises the failure
  count by roughly 72, deliberately. Shares no files with anything else. The dedupe block has no
  test at all today, which is how this slipped through, so add one.
• 2. Exclusions for structurally untestable operations.
  configs/camunda-oca/request-validation.json for createInitialAdminUser and
  restoreAsClusterAdmin, group E. Group D is deliberately not in here.
• 3. Resource fixtures for OCA. configs/camunda-oca/request-validation.json plus a standalone
  make_fixtures step in scripts/e2e/run-oca.sh. Clears 34.
• 4. servers: awareness and cluster-admin credentials. spec/loader.ts, model/types.ts,
  emit/qaEmitter.ts, templates/support/http.ts, templates/support/env.ts,
  scripts/e2e/run-oca.sh. Clears 16. The largest change of the set. Find the credentials first.
• 6. knownIssues entries for F through K, referencing the upstream issues below.

Ordering constraints

Two hard dependencies, and that is all. 0b needs 0, because the whole point of building the
workflows first is that the baseline runs in CI. 6 needs 5, because 5 adds roughly 72 group F
failures, and the group F knownIssue entry should describe them as one thing rather than two.
Everything else is soft. Items 1 through 5 need 0b only to have a number to compare against.

Two files attract more than one sub-issue and want serialising. None of them need combining, though.
Each stays independently reviewable and testable.

1. configs/camunda-oca/request-validation.json, touched by 2 (excludeOperations), 3
   (resourceFixtures, pathResourceFixtures) and 6 (knownIssues). They write different top-level
   keys, so those conflicts are mechanical. The real collision is the $comment, a single string that
   all three will edit. Serialise 2, 3, 6.
2. scripts/e2e/run-oca.sh, touched by 3 for a make_fixtures step plus RV_FIXTURE_* exports in
   run_rv(), and by 4 for the cluster-admin env in that same function. Serialise 3, 4.

Generated output causes no conflicts. .gitignore:4 ignores /generated/.

Affected tests

Sub-issue	Existing tests that change or need rechecking	New tests to add
0, 0b	none	nothing exists for workflow syntax; optional
1	pagination-limit-invalid.test.ts asserts toHaveLength(4) and values === [-99, 0, 10001, 10100] at :200, :213, :225, :257. Becomes length 2 and [-99, 0].	a oneOf case where one branch declares no maximum, asserting the above-maximum mutations get dropped
2, 6	tests/codegen/known-issue-summary-consistency.test.ts	none
3	resource-fixtures-emit.test.ts	an OCA-shaped fixture map case alongside the Hub-shaped ones
4	env-auth-headers.test.ts, query-param-buildurl-slot.test.ts, tests/codegen/materialize-standalone.test.ts, materialize-support.test.ts	base resolution from a path-item servers: override, and clusterAdminHeaders() emission
5	coverage-applicability-wiring.test.ts:58 reads generate.ts as text; confirm it still parses	one asserting two constraint kinds on the same parameter survive dedupe

Upstream filings on camunda/camunda

• Group B: OffsetPagination.limit is missing maximum: 10000, which the other three oneOf
  branches all declare. Looks like a plain omission. If upstream adds it, the dropped scenarios become
  correct and can come back.
• Group B: {"page":{"limit":10001}} returns 500 on Elasticsearch, "The search server was unable to process the request", which is the max_result_window limit, for a request the contract permits.
• Group F: path-param pattern and maxLength not enforced.
• Group G: page.limit: 0 accepted despite minimum: 1 on every oneOf branch.
• Group H: operationReference minimum: 1 not enforced.
• Group I: blank and whitespace tenantId accepted.
• Group J: 500 on backup operations under rdbms secondary storage. Confirm after the re-baseline.
• Group K: undecodable cursor accepted by searchProcessDefinitionVariableNames.

When this is done

I cannot give a target number, because sub-issue 0b replaces the baseline. The shape to aim for is that
every remaining failure is an API conformance gap carrying a knownIssue link, and no failure comes
from the generator asserting something the contract does not say, or from the environment missing a
route. Groups A through E reach zero. Groups F through K stay red until upstream moves, and the count
climbs by roughly 72 when sub-issue 5 lands.

The nightly reports that number every day and fails while doing it. A steady count means nothing
regressed. A change means something did, and the step summary names the group.

Start by reading docker/docker-compose.yml and scripts/e2e/run-oca.sh, then establish a reproducible unsecured baseline before relying on the dated report. Review the failure taxonomy and the referenced request-validation files, including loader.ts, paginationLimit.ts, config.ts, and qaEmitter.ts. Done means the deployment is measured consistently and the parent causes are split into independently tracked generator, environment, and API work.