JuliaGenAI / JuliaGenAI/DocsScraper.jl
Make a cache of robots.txt
Open
Nobody has claimed this yet.
- Dominant language
- Julia
- Stars
- 13
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Currently the crawler pings robots.txt everytime, even for repeated URLs. For every hostname, make a cache of allowed and disallowed paths to make the crawling faster. Speed increase: 25-30%
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the crawler code that requests robots.txt and trace how repeated URLs are handled. Implement the per-host cache for allowed and disallowed paths, then verify that repeated URLs avoid redundant robots.txt requests while preserving crawling behavior and targeting the stated speed improvement.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- julia
- Domain
- backend, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100