dotCMS / dotCMS/core

tech-debt: batch gather-release-data lookups over GraphQL (538 requests -> 13, and fix Closes: #N)

Open
#35,763 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Team : Enablement Type : Task
Dominant language
Java
Stars
970
Forks
486
Avg merge
3d 33m
Merged PRs (30d)
170

Description

Background

.github/scripts/gather-release-data/ talks to GitHub over REST, one request at a time, in two hot loops:

Call Requests per full release
resolvePRNumbersGET /commits/{sha}/pulls, per commit 485
fetchPRDetailsGET /pulls/{n}, per PR 51
fetchCommitRange — compare API 2

Plus 35 mandated sleep(500) calls between batches — ~17s of doing nothing — added in #37213 to stay under the secondary rate limit. The @octokit/plugin-throttling dependency exists for the same reason.

Under merge commits (no squash) the commit count per release is 5-10x the PR count, so this loop scales with the wrong number.

The correctness half — this is the part that actually hurts

PR details extract linked issues by regex-matching the body (extractLinkedIssues):

/(?:close[sd]?|fix(?:e[sd])?|resolve[sd]?)\s+#(\d+)/gi

\s+ requires whitespace directly after the keyword. So:

PR body Matched?
Closes #123
Closes: #123
Fixes: #123
closes dotCMS/core#123
Resolves: https://github.com/dotCMS/core/issues/123
linked only via the Development sidebar

Closes: #NNN — with the colon — is the form CLAUDE.md mandates for every dotCMS PR, because the link-issue merge gate requires it. So the house style is precisely the form that doesn't match.

Measured on v26.08.24-01...v26.09.02-01 (51 PRs), against GraphQL closingIssuesReferences as ground truth:

  • 23 of 51 PRs (45%) lose their issue link entirely.
  • 27 issue refs found vs 50 actually present.

Symptom on the published release: bullets in the changelog missing their (#N) cross-link. Categorization and rollback-unsafe detection are unaffected.

Proposed change

Replace both loops with GraphQL alias batching — 50 commits/query via object(oid:) { ... on Commit { associatedPullRequests } }, 20 PRs/query via pullRequest(number:), pulling title, body, labels and closingIssuesReferences in the same round-trip.

before:  485 + 51 + 2  = 538 requests + ~17s of sleeps
after:    10 +  3 + 2  =  15 requests, no sleeps

closingIssuesReferences is the same relationship GitHub itself renders, so all six rows in the table above resolve natively and the regex can be deleted.

Reference implementation already in the repo: fetchClosingIssueRefs in .github/scripts/release-qa-status/src/github.ts (added in #35762) batches exactly this way.

Out of scope

  • The compare API stays REST. There is no clean GraphQL equivalent for a merge-base commit range with pagination.
  • listStandardReleaseTags stays REST, so @octokit/plugin-throttling stays too — it still paginates the full release list, which is dozens of requests on a repo with this many CLI and LTS tags.
  • Cross-repo closing refs are discarded. The changelog links issues in this repo only.
  • No change to the prompt template or to Change.linkedIssues (still number[]).

Note

This supersedes the original scope of this issue (linked-issue extraction alone) by folding in the commit→PR batching, since one GraphQL query covers both and splitting them would mean touching the same two functions twice.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in .github/scripts/gather-release-data/ at resolvePRNumbers and fetchPRDetails, then read fetchClosingIssueRefs in .github/scripts/release-qa-status/src/github.ts as the reference batching implementation. Replace the two REST loops with the proposed GraphQL batches while leaving the compare API and tag listing unchanged. Done means release data uses the reduced request pattern and changelog issue links come from closingIssuesReferences, including colon, cross-repository, URL, and Development-sidebar cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
github, graphql, javascript
Domain
api, release, tooling
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.