noindex/nofollow on a per-page basis

Opening an issue, following this [#coderedcms thread](https://wagtailcms.slack.com/archives/CEU45SBRR/p1601038281002900).

## Prerequisites
To understand better how noindex/follow work, [this is a good starting point](https://developers.google.com/search/reference/robots_meta_tag).

## Goal
Ability to control on a per-page basis the presence of `noindex` and `nofollow` robot instructions.

## How do I imagine it?
From the top of my mind, the most needed feature would first to be able to `noindex` a page, in order to have a better control over what part of a website should be indexed by search engines.

The most basic approach would be to have a checkbox in the "SEO" tab, like the following:
- [ ] Exclude this page from being indexed by search engines

or another version for more technical users:
- [ ] "noindex" this page (using "``" tag)

It would then output `` in the head of the page. That's it.

**Edit:** Exclude no-indexed pages from the sitemap. See [this comment](https://github.com/coderedcorp/wagtail-seo/issues/4#issuecomment-732152782).

## Suggestions to make it better
- Let the user choose if the `noindex` directive should be applied via a tag in the , or via request headers. Could be a global setting (site-wide), or local to each page, not sure what would be best.
- Add a similar checkbox to enable `nofollow`
- Apply `noindex` or `nofollow` on the current page, and all child pages too. Useful when for example you have a pure SEA marketing group of pages that you want to be isolated. For instance `www.company.com/lp/`, every child pages could have `noindex` preset, inheriting this setting from `/lp/`.
- A way to control which bots to target (tricky, and quite advanced I guess).

Start by reviewing issue #4, the linked #coderedcms thread, and the Google robots meta tag reference. Define the first supported scope—per-page noindex, nofollow, sitemap behavior, or broader suggestions—and identify the relevant Wagtail SEO entry points. Done means the selected behavior is clearly specified and verified for page output and sitemap inclusion.

Written by the indexing model from the issue text.