Stop google from indexing internal URLs

Today when googling for something Google returned a clojurians-log-internal URL. This bypasses CloudFlare, going to the server directly.

We have `canonical` meta tags on our pages, but somehow Google is still indexing the wrong URL.

Not sure what else we can do about this, but some ideas

- serve a more aggressive `robots.txt` when accessed via -internal
- redirect from internal to the canonical URL if it looks like a search engine crawler
- register the internal domain as well in google search console, maybe that way we get some control over it

The issue names canonical meta tags and proposes robots.txt, crawler-specific redirects, and Google Search Console, but no repository files, tests, or entry points. Start by locating the page URL generation, canonical metadata, and robots handling; done means internal URLs are no longer indexed while canonical public URLs continue to work.

Written by the indexing model from the issue text.