dotCMS / dotCMS/core

Exclude spec-only and documentation-only PRs from the Release QA report

Open
#37,486 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

dotCMS : Build Team : Scout Type : CI/CD
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Description

The release QA report (.github/scripts/release-qa-status) walks every PR merged between two release tags and asks whether its linked issue carries a QA : Passed / QA : Not Needed / QA : Failed label. PRs with no answer land in the missing / unlinked buckets, which drive the :rotating_light: Slack warning on the release announcement and the cc line that @-mentions the authors.

Since Spec-Kit adoption, every feature ships two PRs, and PR 1 carries spec.md alone under specs/<issue>-<slug>/. Those PRs contain nothing runnable, so QA has nothing to exercise and they never earn a QA label — yet the report counts them as un-QA'd. Pure documentation PRs have the same problem.

Impact, measured on v26.09.03-01...v26.09.09-01 (38 PRs): 9 of the 25 missing PRs — 36% of the reported QA gap — were spec- or docs-only. Their authors were @-mentioned on #release-notifications for work QA cannot act on, and the real gaps were diluted by the noise.

The report cannot currently tell the difference: classifyExclusion only ever reads authorType, author, labels and title, and PRDetails has no notion of which files a PR touched. Nothing anywhere under .github/ fetches a PR's changed files.

Acceptance Criteria

Classification

  • A PR whose every changed path is under specs/ is excluded with reason spec-only
  • A PR whose every changed path is documentation but not wholly under specs/ is excluded with reason docs-only
  • Documentation is defined as specs/**, docs/**, and loose *.md / *.mdx anywhere in the tree
  • A PR that changes a spec and implementation (e.g. specs/x/spec.md + .java) is not excluded and is evaluated for QA as before
  • Path matching is case-insensitive (README.MD, SKILL.md, CLAUDE.md all resolve correctly)

Agent tooling stays in QA scope

  • Paths under .claude/, .agents/, .cursor/ and .specify/ are treated as implementation, not documentation
  • Any file named CLAUDE.md or AGENTS.md, at any depth, is treated as implementation
  • A markdown-only PR that ships a Claude skill (e.g. #37309) remains in QA scope
  • The implementation-tooling check runs before the documentation patterns, so .claude/skills/x/SKILL.md cannot match the loose markdown rule

Fail-safe behaviour

  • A PR is excluded only when its changed-file list was retrieved in full and every path matched
  • A PR whose file list is unavailable (API error) stays in QA scope
  • A PR whose file list is truncated (more files than one page returns) stays in QA scope, with a note on stderr naming the PR
  • A failure fetching changed files does not abort the report — the QA section still renders, without path-based exclusion
  • "Unknown file list" is represented distinctly from "empty file list"; neither triggers an exclusion

Rule precedence

  • Path classification runs after the existing author, label and title rules, so a bot-authored spec PR is still reported as bot-author

Reporting

  • Excluded PRs remain visible and auditable in the Excluded section of the text and markdown outputs, showing their reason
  • The Excluded section renders even when no PR is flagged — otherwise a release whose only gaps were spec PRs shows a reduced count with no record of what was skipped
  • The Slack output stays silent when the only non-passing PRs are excluded ones, and stops @-mentioning their authors

Verification

  • Unit tests cover each rule above, including the ordering and fail-safe cases
  • Verified end-to-end against a real release range, with every newly-excluded PR confirmed spec/docs-only through an independent check (not the code under test), and no docs-only PR left in the flagged buckets

Priority

Medium

Additional Context

Scope: CI tooling only — the standalone TypeScript CLI under .github/scripts/release-qa-status/, invoked by .github/workflows/cicd_6-release.yml. No product code and no workflow changes.

Design notes:

  • Changed files are fetched with batched GraphQL (20 PRs per query) rather than one REST pulls.listFiles per PR — the report already spends 3+ REST calls per PR, so this adds roughly one request per twenty rather than one per PR.
  • Unlike the sibling fetchClosingIssueRefs, a failure fetching files is swallowed rather than re-thrown. There, an empty result silently demotes PRs to unlinked and floods Slack, so failing loudly is correct; here an unknown file list simply means no path-based exclusion — the behaviour that existed before this change — and killing the whole QA section over it would be the worse trade.
  • The file list is not paginated past the first page. It exists only to answer "is every file documentation?", and a 100+ file PR never is.

Known side effect: a spec PR whose linked issue already carried QA : Passed moves from the passed bucket to excluded, so passed no longer counts spec PRs that happened to be labelled.

Implemented in dotCMS/core#37484.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in .github/scripts/release-qa-status/, especially classifyExclusion, PRDetails, and the existing fetchClosingIssueRefs flow; inspect the implementation in dotCMS/core#37484. Run the CLI's unit tests and verify changed-file retrieval, precedence, fail-safe handling, output sections, and Slack behavior against the acceptance criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, graphql, typescript
Domain
ci-cd, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.