firecrawl / firecrawl/llmstxt-generator

Feature request: Restrict URLs to a particular folder

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
536
Forks
70
PR merge metrics
No merged PRs in 30d

Description

I wanted to generate the llms txt for "Claude Code". Unfortunately, it spiders into ALL Claude documentation, including discovering URLs for other languages.

Here is the root URL:
https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/overview

I'd love to be able to restrict it to map only the URLs beneath the `https://docs.anthropic.com/en/docs/agents-and-tools/claude-code/*` root URL.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.