allenai / allenai/natural-instructions
preventing search engine crawlers
Open
enhancement
- Dominant language
- Python
- Stars
- 1k
- Forks
- 197
- PR merge metrics
- No merged PRs in 30d
Description
We should add headers to the files so that internet crawlers won't crawl our data.
This will prevent future language models from pre-training on this data.
We may want to get some inspiration from Big-bench on this.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.