camunda / camunda/api-test-generator

Evaluate: let hub-pr-check's classify job open a fix/suppress PR for confirmed generator-gaps

Open
#515 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
0
Forks
3
Avg merge
13h 41m
Merged PRs (30d)
23

Description

Current state: hub-pr-check.yml's classify job is deliberately
read-only — its own comment says it "Mints NO write-capable token for either
repo — it cannot file an issue, open a PR, or push, regardless of the CLI
permission flag below" (hub-pr-check.yml:139-141). When it determines a
failure is generator-gap (api-test-generator doesn't handle a new/changed
endpoint shape yet), that verdict currently only becomes a Slack message —
nothing acts on it.

Why it might be worth doing: triage-camunda-hub-nightly.yml already
does something structurally similar for the nightly run — it mints a
real, write-capable App token and opens fix/suppress PRs in
api-test-generator for issues it finds. hub-pr-check.yml's classify job
runs the exact same kind of analysis, just triggered per-PR instead of
nightly, and per-PR is closer to the moment the gap is actually introduced.

Suggested scope, if built: NOT auto-writing generator logic (understanding
a new schema shape and extending the ontology/scenario-templates is a real
engineering task, not something to ship unreviewed). Start narrower — when
classify confirms generator-gap at high confidence, open a PR that adds
the corresponding excludeOperations/knownIssue entry to the relevant
configs/<config>/request-validation.json (mechanical, low-risk, exactly
what a human currently does by hand — e.g. #512's createWorkspace entry).

Open questions to resolve before building (not resolved by this issue):

  1. Trust/security. hub-pr-check.yml is triggered by repository_dispatch
    from ANY camunda-hub PR — not necessarily reviewed or trusted at the time
    the check runs. classify's read-only posture exists specifically to keep
    its blast radius to "read files, write one JSON output" regardless of what
    content it's fed. Granting it (or a derived job) write access to
    api-test-generator based on an LLM's read of that external PR's content
    adds a real prompt-injection surface — likely needs at least the same
    class of guards already used elsewhere in this repo (the .github/**-diff
    check in triage-camunda-hub-nightly.yml/hub-pr-live-check.yml, a
    narrowly-scoped App token), possibly more caution given the trigger
    originates from a different repo entirely.
  2. Noise/dedup. hub-pr-check.yml runs on every camunda-hub PR — per the
    recent investigation (#514), a single standing issue can trigger a burst
    of a dozen+ runs in under an hour across many unrelated PRs. Without a
    dedup guardrail (the nightly already has one — "Discover open fix PRs",
    triage-camunda-hub-nightly.yml:47-51's comment on why pull-requests: read is needed for it), this could open many duplicate PRs for the same
    gap.
  3. False positives. generator-gap is itself an LLM guess, now further
    hardened (#514) but still not certain — an auto-opened PR based on a wrong
    guess is more consequential than a Slack message based on one.

Not a commitment to build — filed as a candidate enhancement to evaluate,
following up on the classify-hardening work in #514.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing hub-pr-check.yml with triage-camunda-hub-nightly.yml and hub-pr-live-check.yml, especially their token, .github/**-diff, and open-PR handling. Resolve the trust, deduplication, and false-positive questions before proposing implementation. Done means a documented decision and, if approved, narrowly scoped safeguards for any generated PR.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions
Domain
ci-cd, devops, security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.