django / django/djangoproject.com
Use noindex meta tag or header, not robots.txt, to block untranslated docs pages
- Dominant language
- Python
- Stars
- 2k
- Forks
- 1.1k
- Avg merge
- 2d 4h
- Merged PRs (30d)
- 24
Description
As far as I can tell, we're currently using `robots.txt` to block untranslated docs pages from being indexed by Google. Looking through Search Console, it appears that is not the correct approach and these pages are still being indexed:

That warning links to: https://support.google.com/webmasters/answer/7440203#indexed_though_blocked_by_robots_txt
Which suggests using the `noindex` meta tag or header instead: https://support.google.com/webmasters/answer/93710
Contributor guide
No contributing guide indexed for this repository
Research direction
No file or test is named in the issue. Start by locating how robots.txt blocks untranslated documentation pages and where those pages are rendered; verify the affected URLs use a noindex meta tag or header instead, then check that the indexing behavior is no longer blocked solely by robots.txt.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, web-dev
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100