Custom 404 not working in Pages Router if `app` directory exists

Link to the code that reproduces this issue

https://github.com/ajmnz/next-custom-404-repro

To Reproduce

1. Install deps (npm i), start the dev server (npm run dev) and navigate to localhost:3000
2. Follow the instructions shown

Current vs. Expected behavior

I expected Next.js to detect that I'm working with the Pages Router instead of assuming I'm using the App Router just because there's a directory called app.

When an app directory is present, Next.js will output ○ Compiling /not-found .... If renamed, it will change to ○ Compiling /404 ..., which is the expected behavior.

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Operating System:
  Platform: darwin
  Arch: x64
  Version: Darwin Kernel Version 22.6.0: Wed Oct  4 21:25:26 PDT 2023; root:xnu-8796.141.3.701.17~4/RELEASE_X86_64
Binaries:
  Node: 20.8.0
  npm: 10.1.0
  Yarn: 1.22.10
  pnpm: N/A
Relevant Packages:
  next: 14.0.3
  eslint-config-next: N/A
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.3.2
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

Routing (next/router, next/navigation, next/link)

Additional context

I believe the offending release is between 13.3.4 and 13.5.4 (inclusive). I have an app on 13.3.4 with the same setup described (pages router and a directory called app) and custom 404s are working just fine.

Start with the reproduction at https://github.com/ajmnz/next-custom-404-repro: run npm i, npm run dev, and follow its instructions. Investigate how Next.js selects the 404 or not-found route when both the Pages Router and an app directory exist. Done means a custom Pages Router 404 is selected without renaming the app directory, with coverage for this setup.