camunda / camunda/api-test-generator

request-validation: 29 constraint/param-constraint negative tests false-positive (server returns 200, not 400) — not spec drift

Open
#372 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

OCA
Dominant language
TypeScript
Stars
0
Forks
3
Avg merge
13h 41m
Merged PRs (30d)
23

Description

Summary

In the camunda-oca secured request-validation suite, 29 negative tests assert 400 but the server returns 200 — the malformed value rides on an otherwise-valid request, reaches the backend, and is accepted rather than rejected. These are false-positive failures: the server does not enforce the constraint the spec declares (or enforces it differently), so the generated 400 assertion can never pass against a real backend.

This is distinct from the path-id precondition class (#352) and the string path-param 400/404 class (#363), and from the /v2/setup/user bootstrap 403s (#121) — those are already tracked. This got 200 class has no owning issue.

Proof it is NOT spec-vs-server drift

The suite is generated from a pinned 8.9-era spec; the broker is 8.10.0-SNAPSHOT, so "version drift" was the obvious suspect. Ruled out:

  • Re-fetched the spec at camunda/camunda@main (8.10) and regenerated, then ran the secured suite against the matching 8.10.0-SNAPSHOT broker.
  • The got 200 count was essentially unchanged: 30 → 29 (only one missing-required case dropped). The 16 constraint-violation + 13 param-constraint-violation 200s persisted identically.

So matching the spec to the server does not resolve them — the server simply doesn't enforce these constraints.

The 29 failures — three patterns

1. tenantId constraint-violation (8) — server accepts a tenantId that violates the declared pattern/length:

  • publishMessage (tenantId)
  • broadcastSignal (tenantId)
  • getProcessDefinitionInstanceVersionStatistics (filter.tenantId, 4 variants)

2. operationReference out-of-range constraint-violation (8) — server does not range-check operationReference on batch operations (variants #2/#3, e.g. 0 / -99):

  • cancelProcessInstancesBatchOperation, deleteProcessInstancesBatchOperation, modifyProcessInstancesBatchOperation, resolveIncidentsBatchOperation

Note vs #352: #352/#124 list operationReference variants too, but for operations with a synthetic path id (e.g. resolveIncident, deleteProcessInstance) those surface as 404 (path-id masks body validation) and are covered there. The batch operations above have no path id, reach validation, and return 200 — a different root cause that belongs here.

3. Path-param length-max violation (13) — an over-long path key is accepted and the search just returns empty instead of 400:

  • search{Clients,Users,Roles,MappingRules,GroupIds,Groups}For{Group,Role,Tenant} (groupId / roleId / tenantId length-max)

Why this matters

These 29 erode trust in the secured suite: they fail on every run against a real backend for reasons unrelated to the code under test. Combined with #352 (404 class) and #121 (403 class), they account for the bulk of the standing secured-suite red.

Proposed direction

Per-pattern triage — each case is either (a) a generated test the suite should stop emitting / not assert 400 for (the server legitimately doesn't constrain that field at that endpoint), or (b) a genuine upstream server gap to file against camunda/camunda. Suggested:

  • Add a precondition/affordance check so constraint-violation / param-constraint-violation scenarios are only emitted for fields the backend actually enforces (mirrors the "don't emit a test the contract doesn't require" theme in #352's Class 3).
  • For fields that should be enforced (e.g. tenantId format, operationReference range), file the corresponding server-side bugs and keep the tests as known-failing references until fixed.

Repro

# 8.10 spec + 8.10 server (secured: UNPROTECTEDAPI=false, basic auth demo/demo)
SPEC_REF=main CONFIG=camunda-oca npm run fetch-spec
CONFIG=camunda-oca npm run generate:request-validation
RV_PROFILE=secured CORE_APPLICATION_URL=http://localhost:8080 \
  CAMUNDA_BASIC_AUTH_USER=demo CAMUNDA_BASIC_AUTH_PASSWORD=demo \
  npm run test:pw:request-validation
# -> 29 tests assert 400, receive 200 (see categories above)

Found while verifying #362 / PR #370 against 8.10.0-SNAPSHOT (full secured-suite triage).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the secured request-validation suite by running the listed fetch-spec, generate:request-validation, and test:pw:request-validation commands against the matching broker. Trace how the 29 tenantId, operationReference, and path-param length-max scenarios are generated. Done means each case is either gated from emitting an invalid 400 assertion or documented as an upstream server gap with the relevant test behavior preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
openapi, typescript
Domain
api, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.