Hide preview deployments from search engines

What problem will this feature address?

When a preview deployment is made, it's currently open to being scraped by search engines. I think for most people that would not be a good thing.

Generally, preview deployments are intended to be temporary.

Describe the solution you'd like

It would be nice to have an option to prevent search engine indexing for preview deployments. When enabled, add an X-Robots-Tag header with a value of noindex,nofollow to the preview deployments traefik setup.

Describe alternatives you've considered

As a workaround for anyone else who has this requirement you can manually add the header into the traefik yml files. It's manual, but it works.

Follow these steps after creating your preview deployment setup. Then reload Traefik from the Web Server menu.

No-follow middleware (one-time setup)

• Select Traefik File System from the main menu.
• Open the dynamic folder, select the middlewares.yml.
• Add the following middleware:

  http:            # should already exist
    middlewares:   # should already exist
      no-search:
        headers:
          customResponseHeaders:
            X-Robots-Tag: noindex,nofollow
  

Add no-follow middleware to service

• Select Traefik File System from the main menu.
• Open the dynamic folder, select the preview config file (something like preview-<app-service>-xyz.yml.
• Add the no-search middleware (created above):

  http:
    routers:
      preview-app-name-router:
        service: preview-app-name-router
        middlewares:
          - redirect-to-https
          - no-search    # Add this line
        entryPoints:
          - web
      preview-app-name-websecure:
        service: preview-app-name-websecure
        middlewares:
          - no-search    # Add this line
        entryPoints:
          - websecure
  

Additional context

No response

Will you send a PR to implement it?

Maybe, need help

Start by tracing how preview deployments generate their Traefik configuration, including the dynamic preview config and middleware definitions described in the issue. Determine where an indexing option would be handled and verify that enabled previews receive the requested X-Robots-Tag header in both HTTP and HTTPS routes.