pingdotgg / pingdotgg/t3code

Unrecognised GitHub Enterprise host re-probes fj, tea and glab once per project on every sweep

Open
#12,060 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

accepted bug via-triage
Dominant language
TypeScript
Stars
23k
Forks
5.9k
Avg merge
11h 14m
Merged PRs (30d)
357

Description

What happened

Threads open slowly and the VCS/PR state in the sidebar takes 10–18s to settle. Not tied to a
particular update — it has always behaved this way on this machine. The desktop app intermittently
shows "Reconnecting" whenever anything else on the box uses CPU.

Every repository I work on is hosted on a GitHub Enterprise Cloud tenant with a data-residency
hostname of the form <tenant>.ghe.com. gh is authenticated against that host and works fine
from the terminal.

Diagnosis

A GHE host is classified unknown, can never be refined, and the failed refinement is retried on
every read without ever being cached. Verified against main @ ccf220b.

1. <tenant>.ghe.com is never detected as GitHub.

packages/shared/src/sourceControl.ts:183 matches GitHub as host === "github.com" or a github
DNS label. <tenant>.ghe.com has neither, so detectSourceControlProviderFromRemoteUrl returns
kind: "unknown".

2. Nothing can refine it. Of the five discovery specs, only GitLab
(GitLabSourceControlProvider.ts:107) and Forgejo (managed-cli) register a refineUnknownRemote.
The GitHub spec registers none. So there is no code path by which a GHE host can ever resolve to
github, no matter how many times it is probed.

3. The refinement runs uncached on every read. SourceControlProviderRegistry.ts:261 only
consults providerContextCache when context is undefined. Both hot callers pass context
explicitly, so both bypass it entirely:

  • PullRequestService.ts:615 (refineUnknownProjectKinds) — no caching whatsoever
  • server.ts:307 (identity resolver refine) — 1 min TTL

Negative results are discarded rather than cached: ForgejoSourceControlProvider.ts:150
(orElseSucceed(() => [])), SourceControlProviderDiscovery.ts:328 (orElseSucceed(() => null)),
and the registry cache uses Duration.zero on failure (SourceControlProviderRegistry.ts:258).

4. firstSuccessOf multiplies it by the number of projects on the host.
refineUnknownProjectKinds groups candidates by baseUrl and runs Effect.firstSuccessOf over
them. Every project on the same GHE host lands in one group, and because none can ever succeed, the
entire list is exhausted on every call. 11 of my 14 projects share one host.

Each candidate spawns fj, tea and glab. None are installed — these are pure failed spawns.

Measured: one PullRequestService.canonicalRef = 12 refinements → 24 listLogins → 20.8s.
PullRequestSyncReactor runs syncGroup at concurrency: 8, every 60s.

5. WSL2 makes each failed spawn ~100x more expensive. This is why it is so much worse here than
it would be on native Linux. The server's PATH contains 36 /mnt/c/... entries, so every ENOENT
lookup is a scan across the 9p filesystem. Measured on this machine, 20 spawns of a missing binary:
3.05s (152ms each) versus 0.043s (2.1ms each) with a stub resolving first on PATH.

The result is a sweep that cannot keep up with its own 60s schedule, and a server with no CPU
headroom — so any unrelated load (a test run) pushes the health endpoint past the desktop app's 10s
timeout and triggers "Reconnecting".

Steps to reproduce
  1. Have gh authenticated against a GitHub Enterprise host whose hostname contains no github DNS
    label — e.g. <tenant>.ghe.com, or any GHES vanity domain.
  2. Add several projects (I have 11) whose origin remote points at that host.
  3. Do not install fj, tea or glab.
  4. Open the app and watch server.trace.ndjson.

Every PullRequestSyncReactor sweep re-probes all three CLIs once per project, forever. The effect
scales with the number of projects sharing the host, and is far more visible on WSL2 than on native
Linux or macOS because of the PATH cost above.

Version

0.0.41-nightly.20260916.1795 (t3 triage context reports installed version 0.0.42)

Environment

WSL2 (Ubuntu, kernel 6.6.87.2-microsoft-standard-WSL2) on Windows, x64, 16 cores.
Node v26.8.2. Desktop app against a local server. gh 2.x authenticated to both github.com and
the enterprise host. fj, tea, glab, az not installed.

Evidence
225s window, 4 rotated trace files, 21,847 spans

  n     p50        max        span
  1629  335.0ms    2674.6ms   ForgejoCli.listLogins        (874 failures, ~50%)
  816   1175.8ms   4341.6ms   refineUnknownRemoteProvider
  815   658.4ms    2718.3ms   ForgejoSourceControlProvider.refineUnknownRemote
  1762  476.2ms    3641.2ms   processRunner.runProcessCore

spawn failures in window:  glab 881,  tea 873

slowest spans:
  119405ms  PullRequestSyncReactor.sweep         (on a 60s Schedule.spaced)
   31316ms  ws.rpc.vcs.refreshStatus
   30127ms  ws.rpc.subscribeVcsStatus
   23001ms  PullRequestSyncReactor.syncGroup     (x8 back-to-back, concurrency: 8)
   23001ms  PullRequestService.canonicalRef

one canonicalRef subtree (20.8s):
   12  refineUnknownRemoteProvider
   12  ForgejoSourceControlProvider.refineUnknownRemote
   12  ForgejoCli.readKeys
   24  ForgejoCli.listLogins
   24  processRunner.runProcessCore

glab spawn parentage (796 of 881):
   VcsProcess.runUnbounded < VcsProcess.run < refineUnknownRemoteProvider < PullRequestService.canonicalRef
Related issues

#5087 (GHE support, closed; its PR #6052 closed unmerged) and #5089 (still open) would both remove
the root cause. This issue is about the cost of the failure mode itself: even with GHE detection
shipped, any genuinely unrecognised host still re-probes three CLIs per project per sweep forever,
because the negative result is never cached and firstSuccessOf exhausts the whole candidate group.
#11220 is adjacent — idle sweep cadence and caches that expire before the next sweep — but its root
cause is scheduling, not unrefinable provider detection. #5932 is the change that made hostname
matching deliberately strict.

Fix applied or workaround

No fix for the detection itself; the playbook rules out patching the source. As a local mitigation,
shadowed the three never-installed CLIs so the lookups stop scanning the Windows PATH:

for c in tea fj glab; do
  printf '#!/bin/sh\nexit 1\n' > ~/.local/bin/$c
  chmod +x ~/.local/bin/$c
done

VcsProcess.ts:167 fails on non-zero exit when allowNonZeroExit is unset, so discovery still
reports both providers as missing and refinement still returns unknown — identical behaviour,
cheaper. Measured before/after:

ForgejoCli.listLogins p50   335ms  ->  187ms
canonicalRef p50           20.8s   ->  10.9s
syncGroup max              23.0s   ->  10.9s

Roughly 2x end-to-end — less than the isolated PATH benchmark suggests, because the spans also
include spawn overhead and 8-way contention. It is a mitigation, not a fix: the probe storm is
still there, just cheaper per probe.

Filed by

claude (opus-5) via t3 triage

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with packages/shared/src/sourceControl.ts, SourceControlProviderRegistry.ts, PullRequestService.ts, server.ts, and the discovery provider files named in the report. Trace how explicit contexts bypass providerContextCache and how failed refinement results are handled. Done means an unrecognised host no longer triggers repeated CLI probes for every project on each sweep, with regression coverage for the negative result and cache behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
backend, devtools, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.