CodeForAfrica / CodeForAfrica/ui
@Roboshield update list of crawlers to block on robots.txt
Open
enhancement
- Dominant language
- JavaScript
- Stars
- 3
- Forks
- 2
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 5
Description
Comparing to other sites, we may be missing some crawlers
[The Guardian](https://www.theguardian.com/robots.txt), [Washington Post](https://www.washingtonpost.com/robots.txt), [BBC](https://www.bbc.com/robots.txt), etc.
_Originally posted by @kilemensi in https://github.com/CodeForAfrica/ckanext-openafrica/pull/55#discussion_r1908383134_
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.