BuilderIO / BuilderIO/gpt-crawler
How to search sub-folders e.g. xyz.com/folder/page1, page 2 etc.
Open
- Dominant language
- TypeScript
- Stars
- 22.4k
- Forks
- 2.4k
- PR merge metrics
- No merged PRs in 30d
Description
How do you search for all the sub folders. e.g.
https://thoughtcatalog.com/trisha-bartle/
https://thoughtcatalog.com/trisha-bartle/page/2/
https://thoughtcatalog.com/trisha-bartle/page/3/
etc.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named; start by locating the crawler's URL discovery and traversal entry points. Use the example URL and its /page/2/ and /page/3/ variants to determine the expected behavior, with completion defined as crawling the subfolder's paginated pages.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100