Handling robots.txt on Workers
- Dominant language
- TypeScript
- Stars
- 11
- Forks
- 14
- PR merge metrics
- No merged PRs in 30d
Description
to prevent spawning /robots.txt on workers (that are decreasing out SEO score)
Its a must to either do
1. separate (proxy) worker
2. for each worker add handler for robots.txt
READING:
- https://community.cloudflare.com/t/worker-needed-for-robot-txt-value/282993
- https://www.cloudflare.com/en-gb/learning/bots/what-is-robots-txt/
- https://til.simonwillison.net/cloudflare/robots-txt-cloudflare-workers
Contributor guide
No contributing guide indexed for this repository
Research direction
No file, test, or entry point is named in the issue. Start by reviewing the linked Cloudflare robots.txt readings and locating the relevant Worker entry points; done means preventing unwanted /robots.txt handling through one of the two proposed approaches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100