coreruleset / coreruleset/plugin-registry
Rules for AI Crawlers
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 53
- Forks
- 12
- Avg merge
- 7h 30m
- Merged PRs (30d)
- 1
Description
Motivation
AI Crawlers are a massive issue on the web nowadays. Bad acting ones, which disrespect robots.txt, normally act basically as a DDoS on the application/website.
Proposed solution
I'd like to suggest adding new rules, specifically targeted at detecting malicous crawlers(through more than just user agent, as that's easy to lie about) in order to better protect against such cases.
Alternatives
I've considered trying tools such as https://github.com/fabriziosalmi/patterns that add on top of OWASP but I think it'd be best if this project had it
Additional context
AI Crawlers have been a huge problem, specially for smaller websites. Most of those rely on OWASP for defense and things like cloudflare.
Cloudflare however is extremely bad/basic, specially on the free tier, so it'd be great if OWASP did some level of bot/ai crawler protection
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the plugin registry and the OWASP CRS context referenced in the issue, then review the linked patterns project for comparison. The issue does not name files, tests, detection signals, or acceptance criteria, so those would need to be defined before implementation and validation.
Written by the indexing model from the issue text.
Assessment
- Domain
- security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100