messense / messense/robotparser-rs
Sitemap (and perhaps other directive) behavior
Open
- Dominant language
- Rust
- Stars
- 18
- Forks
- 10
- PR merge metrics
- No merged PRs in 30d
Description
In the wild Sitemap directives are overwhelmingly placed at the root of the document and are not associated with a User-agent. Currently Sitemap directives not associated with a User-agent are dropped silently.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the robots.txt parsing logic that associates directives with User-agent records, then trace how Sitemap directives at the document root are handled. Confirm the expected behavior for root-level Sitemap entries and add coverage showing they are retained rather than silently dropped.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- web-dev
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100