Segment cache: prefetched sibling routes render the wrong page when `htmlLimitedBots` disables PPR serving (16.3.3; no longer reproduces on 16.4.0-canary.0)

Link to the code that reproduces this issue

https://github.com/nlindema/segment-cache-shell-poisoning-repro

To Reproduce

1. npm install, npm run build, npm run start (production — the bug needs the prerender manifest; it is not a next dev scenario)
2. Open / in a fresh browser profile (the poisoned cache entry persists for the session once written)
3. Wait ~3 seconds for the four /item/* link prefetches to settle
4. Click alpha

The app is five small files: next.config.js with cacheComponents: true, partialPrefetching: true, and htmlLimitedBots: /.*/ (blocking metadata for every UA — an SEO policy choice; this is what arms the bug), a static listing page with four <Link>s to /item/alpha … /item/delta, and app/item/[...slug]/page.js where a 'use cache' data read feeds both the page body and generateMetadata. The build emits /item/[...slug] as ◐ (Partial Prerender).

Current vs. Expected behavior

Current (16.3.3, 3/3 across fresh browser contexts): the URL is /item/alpha, but <title> and <h1> render Item delta (or Item charlie — whichever sibling happened to be prefetched last). The wrong content is never corrected after render. All four /item/* prefetch responses arrive without x-nextjs-postponed. Direct loads and reloads are always correct; only soft navigation is affected.

Expected: navigating to /item/alpha renders Item alpha, regardless of which sibling routes were prefetched.

Control (same app, one line): comment out htmlLimitedBots: /.*/, rebuild — every prefetch response now carries x-nextjs-postponed: 2 and the click is always correct. So the trigger is the htmlLimitedBots catch-all disabling PPR shell serving; the analysis below is why that turns into cross-URL cache poisoning on the client.

Canary: does not reproduce on 16.4.0-canary.0 (or .10) — prefetches still fire without x-nextjs-postponed, but the navigation issues a dynamic request for the clicked URL and renders correctly, with soft navigation intact. Verified boundary with the same harness: 16.3.3 = red 2/2, 16.4.0-canary.0 = green. The range v16.3.3...v16.4.0-canary.0 is 293 commits; candidates by title include #96827 ("Use Cache Components as the internal PPR signal"), #96406 ("Unify RouteTree and CacheNodeSeedData on the client") and #97128 ("Fix: Optimistic routing bugs leading to repeated prefetch loops"). Since 16.3.3 is the latest stable, this is effectively a request to confirm the fix was intentional and consider a 16.3.x backport.

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 25.5.0: Tue Jun  9 22:28:29 PDT 2026; root:xnu-12377.121.10~1/RELEASE_ARM64_T6030
  Available memory (MB): 36864
  Available CPU cores: 12
Binaries:
  Node: 26.4.0
  npm: 11.17.0
  Yarn: N/A
  pnpm: 11.13.1
Relevant Packages:
  next: 16.3.3 // Latest available version is detected (16.3.3).
  eslint-config-next: N/A
  react: 19.2.8
  react-dom: 19.2.8
  typescript: N/A
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

cacheComponents, Partial Prerendering (PPR), Linking and Navigating, Use Cache

Which stage(s) are affected? (Select all that apply)

next start (local), Other (Deployed), next dev (local)

Additional context

Mechanism, from instrumenting the 16.3.3 client segment cache (why the catch-all turns into cross-URL poisoning):

1. htmlLimitedBots becomes a prerender-manifest bypass rule for PPR routes (build/index.js, bypassFor: [{ key: 'user-agent', value: … }]); with /.*/ every request bypasses the shell, including RSC prefetches, which are then served by the legacy dynamic flow (full concrete page, no x-nextjs-postponed: 2).
2. The client scheduler still picks FetchStrategy.PPR — the SubtreeHasPartialPrefetching hint branch wins before route.supportsPerSegmentPrefetching (false) is consulted (segment-cache/scheduler.js).
3. The Shell phase's RuntimeShell request therefore gets a full concrete page instead of an App Shell, and fulfillEntrySpawnedByRuntimePrefetch re-keys it at tree.shellVaryPath — every non-root param replaced by Fallback (segment-cache/cache.js). Each sibling prefetch overwrites the same param-blind keys (page segment, _head, param layout).
4. Lookups fall through to Fallback entries (cache-map.js, getEntryWithFallbackImpl), so navigating to slug A reads slug B's full page, and nothing triggers a corrective fetch.

Two suggestions, independent of the canary fix:

• The client should not store a response under the param-blind shell vary path when it already knows the route does not support per-segment prefetching (routeIsPPREnabled === false was computed for the same route's /_tree response).
• The htmlLimitedBots bypass guards blocking metadata in the HTML response, but it also applies to RSC segment prefetch requests, which carry no HTML head. Serving per-segment prefetches from the segment cache regardless of the UA bypass would let apps that need blocking metadata for every UA still use Instant Navigations — today the two features are silently mutually exclusive.

Deployment: reproduces locally (next build && next start) and on a Docker output: "standalone" deployment identically; browser-independent (Chromium via Playwright and regular Chrome).

Run the linked reproduction with npm install, npm run build, and npm run start, comparing 16.3.3 with 16.4.0-canary.0. Read build/index.js, segment-cache/scheduler.js, segment-cache/cache.js, and cache-map.js to trace the PPR and fallback keys. Done means confirming whether the canary behavior is an intentional fix and determining whether a 16.3.x backport is appropriate.