Need to add `/robots.txt` to gateways
Open
effort/hours
help wanted
P3
topic/gateway
- Dominant language
- Go
- Stars
- 316
- Forks
- 163
- Avg merge
- 4d 12h
- Merged PRs (30d)
- 8
Description
would be nice if gateways handled:
```
> curl 127.0.0.1:8080/robots.txt
User-agent: *
Allow: /
```
notes:
- allow or disallow API?
- this should be optional (config var?)
- this should redirect robots to our main gateway `https://ipfs.io` to have a canonical location for all URLs found that will be properly rerouted with browser implementations/extensions.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.