Epic: Move away from `pages` folder

Back in 2023/2024 when we were releasing the redesigned Node.js Website, we had to create a custom dynamic router around our infrastructure (Next.js) because pages folder structure is what @nodejs/collaborators were used to work with. This was intentionally done to not make the move to Next.js feel as big as an atomic move as it could be. This was also done to keep DX for existing collaborators simple.

Fast forwarding to now, release blog post generation is getting automated and releasers and other collaborators that have worked with us are used and more familiar with our Next.js environment.

The time has probably come. It's time to move away from our custom next.dynamic.mjs router (including all the bells-and-whistles) and adopt Next.js's built-in MDX, which uses mdx-rs, a Rust-based MDX parser/compiler (https://github.com/web-infra-dev/mdx-rs) and that at the moment also supports custom JS-based Rehype and Remark Plugins.

This work effectively means that:

• Our custom Frontmatter Engine for Layouts will be gone. Frontmatter would still exist, but Layouts are decided by the App router.
  • i.e. /app/blog/layout.tsx, etc.
  • each section (learn, download, etc) use their own layouts.
• Each page (.mdx) file under /pages would be moved under the respective folder under /app/ router
• The use of catch-all segments would effectively be diminished (except for some rare situations such as blog category pages and pagination and the index page (that is also a blog category page)
• Adoption of Next.js's built-in MDX parser/compiler that is much faster and also enables support of some real cool stuff
• Our custom glob/mdx parsing and caching engine will be gone

Some of the benefits:

• Much faster compilation and dev runtime
• 1:1 (or almost 1:1, because there are still some magic routes such as blog categories) file-system to route
• More out of the box Next.js experience and easier to maintain and understand codebase
• Fast-refresh / HMR (Hot-Module-Reload) support for MDX
• Simplification of our OpenNext.js infrastructure. No more need of the hacky node:fs polyfills and whatnot

The cons?

• Poor @avivkeller is going to work overtime on this LOL (/s)

cc @nodejs/web-infra @nodejs/nodejs-website