facebook / facebook/docusaurus

Allow site configuration to not index tag pages

オープン
#10,835 コメント 3 件 リアクション 0 件 担当者 0 名 GitHub で見る
feature status: needs triage
主要言語
TypeScript
スター
66.2k
フォーク
10k
平均マージ
1日 3時間
マージ済み PR(30日)
52

説明

### Have you read the Contributing Guidelines on issues?

- [X] I have read the [Contributing Guidelines on issues](https://github.com/facebook/docusaurus/blob/main/CONTRIBUTING.md#issues).

### Description

### Solution

**Proposed API:**

User experience, add new flag for `DocusaurusConfig` in `docusaurus.config` file such as `deindexTags: true`.

**Proposed changes:**

| Behavior | Changes |
| ------------- | ------------- |
| Tag URLs `` elements will have `rel="noindex nofollow"` attributes on tag lists pages. | Update [`Tag`](https://github.com/facebook/docusaurus/blob/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/Tag/index.tsx) to do he check from siteConfig. |
| Tag list page (root component of the tags list page) will have ``. | Update [`DocTagsListPage`](https://github.com/facebook/docusaurus/tree/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/DocTagsListPage/index.tsx) and [`BlogTagsListPage`](https://github.com/facebook/docusaurus/blob/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/BlogTagsListPage/index.tsx) to add the [`Head`](https://docusaurus.io/docs/markdown-features/head-metadata) with noindex meta. |
| Tag page (root component of the "containing tag X" page) ``. | Update [`DocTagDocListPage`](https://github.com/facebook/docusaurus/blob/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/DocTagDocListPage/index.tsx) and [`BlogTagsPostsPage`](https://github.com/facebook/docusaurus/blob/fcc9e5c56a4092cc9c73f1c054854249ab3de3c5/packages/docusaurus-theme-classic/src/theme/BlogTagsPostsPage/index.tsx) to add the [`Head`](https://docusaurus.io/docs/markdown-features/head-metadata) with noindex meta. |
| Sitemap ignores `/tags**` | Theme sets `sitemap.ignorePatterns:[${tagsBasePath}**]`. |

### Motivation

#### Why

Tag pages are thin/low quality, creating duplicated content.
This leads to search engines scoring the website lower, or indexing tag pages before the specific pages.

Google says [1]:

> Block crawling of duplicate content on your site, or unimportant resources (such as small, frequently used graphics such as icons or logos) that might overload your server with requests. Don't use robots.txt as a mechanism to prevent indexing; use the noindex tag or login requirements for that

I have solved this through wrapping/swizzling list pages, tag components and and custom `sitemap.ignorePatterns` rule in the config file, but it's a lot of workaround and a **best-practice** like this would be appreciated if it came as default.

#### Background

This has lead to me issues with all search engines for [privacylearn.com](https://privacylearn.com) to present open-source scripts pre-launch.

siteliner analysis:

Google indexing status:

Engines include Google, Yandex and Bing where thousands of my pages got de-indexed over time and tags pages took more priority than proper pages.

[1]: https://developers.google.com/search/docs/fundamentals/get-started#robots.txt

### API design

_No response_

### Have you tried building it?

_No response_

### Self-service

- [ ] I'd be willing to contribute this feature to Docusaurus myself.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。