Crawl to prerender routes routes with unknown extension

From https://github.com/unjs/nitro/issues/1378#issuecomment-1792423273 by @Barbapapazes

Also #1067

Currently prerenderer crawler, only follows up to crawl links that either have no extension /foo/bar or a known extension /foo/bar/test.json (json is only known extension).

If we have a link such as /foo.bar it won't be crawled as .bar is assumed as an unknown extension.

This behavior is to avoid wrongly crawling paths that lead to binary extensions. Mainly public assets such as /favicon.ico which itself is going to be in public directory already but also static .html files.

I think we should improve the implementation by allowing any extension for a tags at least (current regex for crawler is too aggressive and scans for any tag with href attribute) and let the logic disallow overriding public asset files and also skip binary responses to followup crawler.

Start with the prerenderer crawler implementation and its current href-matching regex, then review the behavior described in the linked issues and comment. Define how links in a tags with unknown extensions are followed while public assets and binary responses remain excluded; done means routes such as /foo.bar are crawled without incorrectly crawling asset files.