Streaming SSR: make the `isbot` "buffer full HTML for bots" behavior configurable (opt-in / opt-out)

Which project does this relate to?

Router — the logic lives in @tanstack/react-router's SSR server (renderRouterToStream) and is used by defaultStreamHandler / createStartHandler, so Start is affected too.

Summary

renderRouterToStream branches on isbot(User-Agent) with no way to override or disable it (packages/react-router/src/ssr/renderRouterToStream.tsx, and L153–160):

// renderToReadableStream branch
if (isbot(request.headers.get('User-Agent'))) {
  await waitForReadyOrAbort(stream.allReady, request.signal)
}

// renderToPipeableStream branch
...(isbot(request.headers.get('User-Agent'))
  ? { onAllReady() { pipeable.pipe(reactAppPassthrough) } }
  : { onShellReady() { pipeable.pipe(reactAppPassthrough) } })

Any UA isbot matches gets the buffered path (whole document withheld until React's allReady); everyone else gets progressive streaming. Reasonable as a default, but it's undocumented and not configurable — neither defaultStreamHandler nor createStartHandler(...)(defaultStreamHandler) expose a hook, and isbot is a hard dependency.

Why this is a problem

isbot classifies synthetic performance tools as bots — Lighthouse (Chrome-Lighthouse), PageSpeed Insights (pagespeed), WebPageTest (PTST), Pingdom, SpeedCurve. So:

1. Lab audits measure a different code path than real users. Lighthouse/PSI hit the buffered onAllReady branch; real browsers hit onShellReady. The lab score stops representing the streaming experience real users (and field/CrUX) get.
2. TTFB inflation on the buffered path. The response is withheld until the slowest loader resolves, so FCP/LCP/Speed Index in those tools are gated behind full data resolution — frequently scoring worse than reality.
3. No way to pick a different policy. Teams may legitimately want to always stream (their crawlers are JS-capable and they prefer fast TTFB), exclude only perf tools, or supply a custom predicate (custom UA list, verified-crawler header, IP tags).

Current workaround (hacky)

The only opt-out today is wrapping the handler and stripping the UA so isbot returns false:

const streamHandler: typeof defaultStreamHandler = (opts) =>
  defaultStreamHandler({ ...opts, request: stripUserAgent(opts.request) }) // isbot(null) === false

This couples app code to an internal heuristic and breaks silently if detection changes.

Proposed solution

A configurable hook, defaulting to today's behavior (no breaking change). e.g. on renderRouterToStream, surfaced through defaultStreamHandler / createStartHandler:

renderRouterToStream({
  request, router, responseHeaders, children,
  // default: (req) => isbot(req.headers.get('User-Agent'))
  isBot: false, // or a predicate: (req) => isbot(...) && !isPerfTool(req)
})

Alternatives: a narrower waitForAllReady?: boolean | ((request: Request) => boolean), or a router-level SSR option set once. Either way, keep the isbot default but let apps override/disable it.

Additional context

• Source: packages/react-router/src/ssr/renderRouterToStream.tsx (present on main; observed in @tanstack/react-router@1.170.16).
• Happy to open a PR once a maintainer signals the preferred API shape.

Start in packages/react-router/src/ssr/renderRouterToStream.tsx, then trace how defaultStreamHandler and createStartHandler expose it. Compare the renderToReadableStream and renderToPipeableStream branches and determine an API that preserves the current isbot default while allowing an override or opt-out; done means the policy is configurable through the affected handlers without changing default behavior.