firecrawl / firecrawl/llmstxt-generator
Feature request: Restrict URLs to a particular folder
Open
- Dominant language
- TypeScript
- Stars
- 536
- Forks
- 70
- PR merge metrics
- No merged PRs in 30d
Description
I wanted to generate the llms txt for "Claude Code". Unfortunately, it spiders into ALL Claude documentation, including discovering URLs for other languages.
Here is the root URL:
https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview
I'd love to be able to restrict it to map only the URLs beneath the `https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/*` root URL.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.