Customize robots.txt in dockerized Dataverse
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 564
- Avg merge
- 2d 2h
- Merged PRs (30d)
- 29
Description
The default robots.txt installed in the Dataverse docker image disallows all access to the site. It would be helpful to be able to configure the image with a customized robots.txt that is similar to the one we use in production.
This issue was encountered in testing the SPA - we need to test that the json-ld added to the header is accessible to web crawlers.
SPA issue: https://github.com/IQSS/dataverse-frontend/issues/350
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the robots.txt installed by the Dataverse Docker image and the image configuration or build entry point that controls it. Compare the default with the production robots.txt referenced in the issue, and review the linked SPA issue for the JSON-LD crawler-access requirement. Done means the image can be configured with a custom robots.txt instead of disallowing all access.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker
- Domain
- devops
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100