Unknown-provider PR polling still floods server logs; closed PR #5831 had a tested fix

What happened

A headless Linux T3 service kept running, but its service log became dominated by repeated source-control warnings. In one ~21-hour boot-service.log, 32,931 warnings were:

PR lookup failed; keeping last known PR state.
provider: unknown
providerOperation: listChangeRequests
errorDetail: No unknown source control provider is registered.

The same condition also produced 1,372 automatic thread settlement skipped warnings with a full cause/stack. Real warnings became difficult to find among the repeated entries.

The affected project uses a self-hosted GitLab remote over SSH with a custom hostname that does not itself identify the hosting provider. T3 resolved the provider as unknown.

The affected path is still present in stable 0.0.38, nightlies 0.0.39-nightly.20260902.1252 and 0.0.39-nightly.20260902.1253, and public main at b520120cf169ce63a5606447a292451e97622c9a.

Diagnosis

This does not appear to be a missing-backoff bug in the provider request itself.

This behavior had a direct proposed fix in #5831, fix(server): skip PR lookup for unknown providers. That PR introduced a typed ProviderUnknown result, avoided calling listChangeRequests for unresolved providers, preserved the last-known PR, and retained retry/backoff so provider refinement could recover. It included focused tests but was closed without merging on Aug 29. Current releases and current main still use the pre-fix error path.

The affected releases already implement per-PR-cache-key exponential failure backoff: 20 seconds, 40 seconds, 80 seconds, and so on, capped at 15 minutes. However, lookupStatusPr calls Cache.get(...) and attaches PR lookup failed; keeping last known PR state. in a catch outside that cache read. A cached failure therefore appears to be logged again for every caller even when the cache prevents a new provider request.

Separately, ThreadSettlementReactor runs every minute. When the same lookup failure reaches it, it emits automatic thread settlement skipped with Cause.pretty(cause). This creates a second repeated warning family for the same durable condition.

Relevant source, pinned to the current reviewed main commit:

• Failure TTL/backoff: https://github.com/pingdotgg/t3code/blob/b520120cf169ce63a5606447a292451e97622c9a/apps/server/src/git/GitManager.ts#L123-L143
• Backoff test: https://github.com/pingdotgg/t3code/blob/b520120cf169ce63a5606447a292451e97622c9a/apps/server/src/git/GitManager.test.ts#L1503-L1510
• Failure-aware cache: https://github.com/pingdotgg/t3code/blob/b520120cf169ce63a5606447a292451e97622c9a/apps/server/src/git/GitManager.ts#L971-L1044
• Per-caller warning after Cache.get: https://github.com/pingdotgg/t3code/blob/b520120cf169ce63a5606447a292451e97622c9a/apps/server/src/git/GitManager.ts#L1104-L1167
• One-minute settlement sweep and warning: https://github.com/pingdotgg/t3code/blob/b520120cf169ce63a5606447a292451e97622c9a/apps/server/src/orchestration/ThreadSettlementReactor.ts#L41-L165
• Closed unmerged fix: https://github.com/pingdotgg/t3code/pull/5831

An unknown provider may be expected for a neutral custom hostname when no source-control CLI reports authentication for that exact host. Regardless of why detection remains unknown, background status and settlement work should treat that durable unsupported-provider state cheaply and should not emit tens of thousands of duplicate warnings.

Possible fix directions:

• Revive or adapt #5831's tested ProviderUnknown outcome so background PR lookup does not treat an unresolved provider as an operational failure.
• Emit the PR-lookup warning only when the cache loader performs a real retry, rather than whenever a caller receives a cached failure.
• Treat provider: unknown as a typed unsupported/no-PR result for background polling, while preserving an actionable error for explicit user operations such as creating a PR.
• Deduplicate or rate-limit automatic thread settlement skipped by project/branch/cause.

Steps to reproduce

1. Configure a repository remote with a neutral self-hosted hostname, for example git@scm.example:group/repo.git, and ensure no provider CLI reports an authenticated provider for that exact host. T3 should resolve the provider as unknown.
2. Run t3 serve and connect a client so VCS status is polled.
3. Have one or more active, unarchived, unsettled threads with branches in that project; leave automatic settlement enabled.
4. Leave the service running for at least ten minutes.
5. Count PR lookup failed; keeping last known PR state. and automatic thread settlement skipped in the service log.

Expected: an unsupported provider is skipped without repeated background warnings, or warnings occur only when the existing backoff performs a real retry.

Actual: repeated callers log the same cached SourceControlProviderError, and the one-minute settlement sweep adds another repeated full-cause warning.

Version

0.0.38-nightly.20260901.1250

Environment

Linux x64 (CachyOS, kernel 7.2.2), Node 26, t3 serve under a systemd user service

Evidence

Aggregate from one ~21-hour boot-service.log (mixed 0.0.33/nightly interval):

32,931  WARN  PR lookup failed; keeping last known PR state.
 1,372  WARN  automatic thread settlement skipped

Representative nightly entry:

[23:55:39.535] WARN: automatic thread settlement skipped
  threadIds: [ '<redacted>' ]
  cause: SourceControlProviderError: Source control provider unknown failed in
    listChangeRequests: No unknown source control provider is registered.
    at Object.listChangeRequests (.../t3/dist/bin.mjs:86505:34)
    at findLatestPrForHeadContext (.../t3/dist/bin.mjs:87300:20)
    at branchPullRequest (.../t3/dist/bin.mjs:87791:28)
    at ThreadSettlementReactor.pullRequestFor (.../t3/dist/bin.mjs:179752:31)
    at ThreadSettlementReactor.sweep (.../t3/dist/bin.mjs:179782:50)


The trace files rotated through about 105 MB during the same day, but this report does not claim that all or most of that volume came from these warnings without an event-type breakdown.

Related issues

• PR #5831 is the exact attempted behavioral fix: it made an unknown provider a typed non-error outcome, skipped listChangeRequests, retained provider-refinement retry/backoff, and preserved last-known PR state. It was closed without merging, so this report supplies current production evidence and asks that the fix be revived or adapted.
• PR #5673 introduced the existing provider-request failure backoff. It reduces actual retry frequency but does not make an unresolved provider a non-error result and therefore does not prevent the repeated warning paths described here.
• #3648 has the exact No unknown source control provider is registered error, but its trigger was an SCP-style remote parser that only accepted the git@ user. That parser was fixed. This report is about repeated logging/caller behavior after a provider remains legitimately or otherwise unresolved.
• #5932 fixed false-positive hostname classification. It is the inverse detection case and does not cover repeated background logging for unknown.
• #6417 and #9057 concern settlement semantics and stale notifications, not cached provider-failure warning volume.
• PR #9125 may share more PR-cache entries between the sidebar and settlement paths, but it does not move/suppress the warning emitted when callers receive a cached failure.

Fix applied or workaround

No application files or state were modified.

If the custom host is GitLab, configuring and authenticating glab for the exact remote hostname may let T3 refine unknown to gitlab and avoid this particular error. That is an environment workaround, not a fix for repeated warning emission when a provider is unsupported.

Filed by

by Codex GPT-5.6 Sol reviewing a GLM-5.3-Flash report

Start with apps/server/src/git/GitManager.ts, especially lookupStatusPr, the failure-aware cache, and the backoff test in GitManager.test.ts. Then read ThreadSettlementReactor.ts and compare the closed PR #5831. Done means unknown-provider polling and settlement no longer produce repeated cached-failure warnings while existing backoff and last-known PR behavior remain covered by tests.