camunda / camunda/api-test-generator
Evaluate: let hub-pr-check's classify job open a fix/suppress PR for confirmed generator-gaps
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 0
- Forks
- 3
- Avg merge
- 13h 41m
- Merged PRs (30d)
- 23
Description
Current state: hub-pr-check.yml's classify job is deliberately
read-only — its own comment says it "Mints NO write-capable token for either
repo — it cannot file an issue, open a PR, or push, regardless of the CLI
permission flag below" (hub-pr-check.yml:139-141). When it determines a
failure is generator-gap (api-test-generator doesn't handle a new/changed
endpoint shape yet), that verdict currently only becomes a Slack message —
nothing acts on it.
Why it might be worth doing: triage-camunda-hub-nightly.yml already
does something structurally similar for the nightly run — it mints a
real, write-capable App token and opens fix/suppress PRs in
api-test-generator for issues it finds. hub-pr-check.yml's classify job
runs the exact same kind of analysis, just triggered per-PR instead of
nightly, and per-PR is closer to the moment the gap is actually introduced.
Suggested scope, if built: NOT auto-writing generator logic (understanding
a new schema shape and extending the ontology/scenario-templates is a real
engineering task, not something to ship unreviewed). Start narrower — when
classify confirms generator-gap at high confidence, open a PR that adds
the corresponding excludeOperations/knownIssue entry to the relevant
configs/<config>/request-validation.json (mechanical, low-risk, exactly
what a human currently does by hand — e.g. #512's createWorkspace entry).
Open questions to resolve before building (not resolved by this issue):
- Trust/security.
hub-pr-check.ymlis triggered byrepository_dispatch
from ANY camunda-hub PR — not necessarily reviewed or trusted at the time
the check runs.classify's read-only posture exists specifically to keep
its blast radius to "read files, write one JSON output" regardless of what
content it's fed. Granting it (or a derived job) write access to
api-test-generator based on an LLM's read of that external PR's content
adds a real prompt-injection surface — likely needs at least the same
class of guards already used elsewhere in this repo (the.github/**-diff
check intriage-camunda-hub-nightly.yml/hub-pr-live-check.yml, a
narrowly-scoped App token), possibly more caution given the trigger
originates from a different repo entirely. - Noise/dedup.
hub-pr-check.ymlruns on every camunda-hub PR — per the
recent investigation (#514), a single standing issue can trigger a burst
of a dozen+ runs in under an hour across many unrelated PRs. Without a
dedup guardrail (the nightly already has one — "Discover open fix PRs",
triage-camunda-hub-nightly.yml:47-51's comment on whypull-requests: readis needed for it), this could open many duplicate PRs for the same
gap. - False positives.
generator-gapis itself an LLM guess, now further
hardened (#514) but still not certain — an auto-opened PR based on a wrong
guess is more consequential than a Slack message based on one.
Not a commitment to build — filed as a candidate enhancement to evaluate,
following up on the classify-hardening work in #514.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by comparing hub-pr-check.yml with triage-camunda-hub-nightly.yml and hub-pr-live-check.yml, especially their token, .github/**-diff, and open-PR handling. Resolve the trust, deduplication, and false-positive questions before proposing implementation. Done means a documented decision and, if approved, narrowly scoped safeguards for any generated PR.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- ci-cd, devops, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100