camunda / camunda/api-test-generator
request-validation: 29 constraint/param-constraint negative tests false-positive (server returns 200, not 400) — not spec drift
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 3
- Avg merge
- 13h 41m
- Merged PRs (30d)
- 23
Description
Summary
In the camunda-oca secured request-validation suite, 29 negative tests assert 400 but the server returns 200 — the malformed value rides on an otherwise-valid request, reaches the backend, and is accepted rather than rejected. These are false-positive failures: the server does not enforce the constraint the spec declares (or enforces it differently), so the generated 400 assertion can never pass against a real backend.
This is distinct from the path-id precondition class (#352) and the string path-param 400/404 class (#363), and from the /v2/setup/user bootstrap 403s (#121) — those are already tracked. This got 200 class has no owning issue.
Proof it is NOT spec-vs-server drift
The suite is generated from a pinned 8.9-era spec; the broker is 8.10.0-SNAPSHOT, so "version drift" was the obvious suspect. Ruled out:
- Re-fetched the spec at
camunda/camunda@main(8.10) and regenerated, then ran the secured suite against the matching 8.10.0-SNAPSHOT broker. - The
got 200count was essentially unchanged: 30 → 29 (only onemissing-requiredcase dropped). The 16constraint-violation+ 13param-constraint-violation200s persisted identically.
So matching the spec to the server does not resolve them — the server simply doesn't enforce these constraints.
The 29 failures — three patterns
1. tenantId constraint-violation (8) — server accepts a tenantId that violates the declared pattern/length:
publishMessage(tenantId)broadcastSignal(tenantId)getProcessDefinitionInstanceVersionStatistics(filter.tenantId, 4 variants)
2. operationReference out-of-range constraint-violation (8) — server does not range-check operationReference on batch operations (variants #2/#3, e.g. 0 / -99):
cancelProcessInstancesBatchOperation,deleteProcessInstancesBatchOperation,modifyProcessInstancesBatchOperation,resolveIncidentsBatchOperation
Note vs #352: #352/#124 list
operationReferencevariants too, but for operations with a synthetic path id (e.g.resolveIncident,deleteProcessInstance) those surface as 404 (path-id masks body validation) and are covered there. The batch operations above have no path id, reach validation, and return 200 — a different root cause that belongs here.
3. Path-param length-max violation (13) — an over-long path key is accepted and the search just returns empty instead of 400:
search{Clients,Users,Roles,MappingRules,GroupIds,Groups}For{Group,Role,Tenant}(groupId / roleId / tenantId length-max)
Why this matters
These 29 erode trust in the secured suite: they fail on every run against a real backend for reasons unrelated to the code under test. Combined with #352 (404 class) and #121 (403 class), they account for the bulk of the standing secured-suite red.
Proposed direction
Per-pattern triage — each case is either (a) a generated test the suite should stop emitting / not assert 400 for (the server legitimately doesn't constrain that field at that endpoint), or (b) a genuine upstream server gap to file against camunda/camunda. Suggested:
- Add a precondition/affordance check so
constraint-violation/param-constraint-violationscenarios are only emitted for fields the backend actually enforces (mirrors the "don't emit a test the contract doesn't require" theme in #352's Class 3). - For fields that should be enforced (e.g.
tenantIdformat,operationReferencerange), file the corresponding server-side bugs and keep the tests as known-failing references until fixed.
Repro
# 8.10 spec + 8.10 server (secured: UNPROTECTEDAPI=false, basic auth demo/demo)
SPEC_REF=main CONFIG=camunda-oca npm run fetch-spec
CONFIG=camunda-oca npm run generate:request-validation
RV_PROFILE=secured CORE_APPLICATION_URL=http://localhost:8080 \
CAMUNDA_BASIC_AUTH_USER=demo CAMUNDA_BASIC_AUTH_PASSWORD=demo \
npm run test:pw:request-validation
# -> 29 tests assert 400, receive 200 (see categories above)
Found while verifying #362 / PR #370 against 8.10.0-SNAPSHOT (full secured-suite triage).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the secured request-validation suite by running the listed fetch-spec, generate:request-validation, and test:pw:request-validation commands against the matching broker. Trace how the 29 tenantId, operationReference, and path-param length-max scenarios are generated. Done means each case is either gated from emitting an invalid 400 assertion or documented as an upstream server gap with the relevant test behavior preserved.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- openapi, typescript
- Domain
- api, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100