django / django/djangoproject.com

Use noindex meta tag or header, not robots.txt, to block untranslated docs pages

Open
#877 8 comments 1 reaction 0 assignees View on GitHub
good first issue
Dominant language
Python
Stars
2k
Forks
1.1k
Avg merge
2d 4h
Merged PRs (30d)
24

Description

As far as I can tell, we're currently using `robots.txt` to block untranslated docs pages from being indexed by Google. Looking through Search Console, it appears that is not the correct approach and these pages are still being indexed:

screenshot 2019-03-02 07 48 44

That warning links to: https://support.google.com/webmasters/answer/7440203#indexed_though_blocked_by_robots_txt

Which suggests using the `noindex` meta tag or header instead: https://support.google.com/webmasters/answer/93710

Contributor guide

No contributing guide indexed for this repository

Research direction

No file or test is named in the issue. Start by locating how robots.txt blocks untranslated documentation pages and where those pages are rendered; verify the affected URLs use a noindex meta tag or header instead, then check that the indexing behavior is no longer blocked solely by robots.txt.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
documentation, web-dev
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.