Allow site configuration to not index tag pages

Abierto

#10,835 abierto el 13/1/2025 3 comentarios 0 reacciones 0 asignados Ver en GitHub

feature status: needs triage

Lenguaje dominante: TypeScript
Estrellas: 66.2k
Forks: 10k
Merge medio: 1 d 3 h
PR fusionados (30 d): 52

Descripción

### Have you read the Contributing Guidelines on issues?

- [X] I have read the [Contributing Guidelines on issues](https://github.com/facebook/docusaurus/blob/main/CONTRIBUTING.md#issues).

### Description

### Solution

**Proposed API:**

User experience, add new flag for `DocusaurusConfig` in `docusaurus.config` file such as `deindexTags: true`.

**Proposed changes:**

| Behavior | Changes |
| ------------- | ------------- |
| Tag URLs `` elements will have `rel="noindex nofollow"` attributes on tag lists pages. | Update [`Tag`](https://github.com/facebook/docusaurus/blob/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/Tag/index.tsx) to do he check from siteConfig. |
| Tag list page (root component of the tags list page) will have ``. | Update [`DocTagsListPage`](https://github.com/facebook/docusaurus/tree/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/DocTagsListPage/index.tsx) and [`BlogTagsListPage`](https://github.com/facebook/docusaurus/blob/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/BlogTagsListPage/index.tsx) to add the [`Head`](https://docusaurus.io/docs/markdown-features/head-metadata) with noindex meta. |
| Tag page (root component of the "containing tag X" page) ``. | Update [`DocTagDocListPage`](https://github.com/facebook/docusaurus/blob/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/DocTagDocListPage/index.tsx) and [`BlogTagsPostsPage`](https://github.com/facebook/docusaurus/blob/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/BlogTagsPostsPage/index.tsx) to add the [`Head`](https://docusaurus.io/docs/markdown-features/head-metadata) with noindex meta. |
| Sitemap ignores `/tags**` | Theme sets `sitemap.ignorePatterns:[${tagsBasePath}**]`. |

### Motivation

#### Why

Tag pages are thin/low quality, creating duplicated content.
This leads to search engines scoring the website lower, or indexing tag pages before the specific pages.

Google says [1]:

> Block crawling of duplicate content on your site, or unimportant resources (such as small, frequently used graphics such as icons or logos) that might overload your server with requests. Don't use robots.txt as a mechanism to prevent indexing; use the noindex tag or login requirements for that

I have solved this through wrapping/swizzling list pages, tag components and and custom `sitemap.ignorePatterns` rule in the config file, but it's a lot of workaround and a **best-practice** like this would be appreciated if it came as default.

#### Background

This has lead to me issues with all search engines for [privacylearn.com](https://privacylearn.com) to present open-source scripts pre-launch.

siteliner analysis: