Next.js does not bundle necessary JSON files in ESM output

Link to the code that reproduces this issue

https://github.com/sofisl/debug-tasks-next.js

To Reproduce

1. Clone the repo listed above,
2. In this line, replace credentials with a credentials key from google cloud platform (following these instructions): https://cloud.google.com/iam/docs/keys-create-delete. Make sure your service account has access to Google Cloud Tasks in your gcp project, and that it is enabled: https://cloud.google.com/iam/docs/manage-access-service-accounts
3. In this line, replace with your project in the format: projects/PROJECT_ID/locations/LOCATION_ID
4. Run npm i && npx next build && node .next/standalone/server.js, and you should see an error.

Let me know if you need help getting a google cloud project/key, I work on the client libraries there and can help with the right debugging.

Current vs. Expected behavior

There is a server error. If it works, you should get the basic Next.js page (after running the create script), and you should be able to listQueues in google cloud tasks in your terminal.

This is the error:

 ✓ Ready in 82ms
 ⨯ Error: Cannot find module '/Users/sofialeon/my-app2/.next/standalone/node_modules/@google-cloud/tasks/build/esm/src/v2/cloud_tasks_client_config.json'
Require stack:
- /Users/sofialeon/my-app2/.next/standalone/node_modules/@google-cloud/tasks/build/esm/src/json-helper.cjs
    at Module._resolveFilename (node:internal/modules/cjs/loader:1082:15)
    at /Users/sofialeon/my-app2/.next/standalone/node_modules/next/dist/server/require-hook.js:55:36
    at Module._load (node:internal/modules/cjs/loader:928:27)
    at Module.require (node:internal/modules/cjs/loader:1149:19)
    at mod.require (/Users/sofialeon/my-app2/.next/standalone/node_modules/next/dist/server/require-hook.js:65:28)
    at require (node:internal/modules/helpers:121:18)
    at getJSON (/Users/sofialeon/my-app2/.next/standalone/node_modules/@google-cloud/tasks/build/esm/src/json-helper.cjs:17:10)
    at file:///Users/sofialeon/my-app2/.next/standalone/node_modules/@google-cloud/tasks/build/esm/src/v2/cloud_tasks_client.js:30:21
    at ModuleJob.run (node:internal/modules/esm/module_job:192:25) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/Users/sofialeon/my-app2/.next/standalone/node_modules/@google-cloud/tasks/build/esm/src/json-helper.cjs'
  ]
}

Provide environment information

Operating System:
  Platform: darwin
  Arch: x64
  Version: Darwin Kernel Version 23.4.0: Wed Feb 21 21:44:31 PST 2024; root:xnu-10063.101.15~2/RELEASE_X86_64
Binaries:
  Node: 20.0.0
  npm: 9.6.4
  Yarn: N/A
  pnpm: 8.6.10
Relevant Packages:
  next: 14.1.0
  eslint-config-next: N/A
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.4.2
Next.js Config:
  output: standalone

Which area(s) are affected? (Select all that apply)

Module resolution (CJS / ESM, module resolving)

Which stage(s) are affected? (Select all that apply)

next build (local), Vercel (Deployed)

Additional context

I work on the Nodejs Client libraries for GCP. We are experimenting with publishing our libraries in dual-format (ESM and CJS), and we just tested this on Cloud Tasks. We got a report that the standalone Next.js build was not working because it was not importing the JSON files: https://github.com/googleapis/google-cloud-node/issues/4858. After reproducing the issue, I thought to fix it by importing the JSON files using require, and wrapping it as a CJS module: https://github.com/googleapis/gapic-generator-typescript/commit/f911ac838d09a1ed0c4e56ea41039a953ed0741c#diff-f90b3a5e1cd374f45c74b508ada7926d1d3daae48c69ff09cac111c349e67d02.

I thought that fixed the issue, but it seems that Next.js still won't copy over our JSON files in the ESM output (however, it will do it in the CJS output), even though both use this [json-helper.cjs.njk](https://github.com/googleapis/gapic-generator-typescript/commit/f911ac838d09a1ed0c4e56ea41039a953ed0741c#diff-f90b3a5e1cd374f45c74b508ada7926d1d3daae48c69ff09cac111c349e67d02) method.

I see this in your docs:

There are some cases in which Next.js might fail to include required files, or might incorrectly include unused files. 

I will ask people to use the config as a workaround, but in the meantime would like to work to get it working without it! Happy to make changes to the google cloud library as well.

Clone the linked reproduction and run npm i && npx next build && node .next/standalone/server.js with output: standalone. Compare the ESM and CJS standalone outputs, focusing on the missing cloud_tasks_client_config.json and json-helper.cjs; done means the standalone server starts and Google Cloud Tasks can list queues without a tracing workaround.