BuilderIO / BuilderIO/gpt-crawler
Crawling duplicated url
Open
- Dominant language
- TypeScript
- Stars
- 22.4k
- Forks
- 2.4k
- PR merge metrics
- No merged PRs in 30d
Description
need to ignore the url if already crawled the page, in my case same url crawled several times
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by locating the crawler entry point and the logic that records or checks visited URLs. Reproduce the repeated crawl described in the issue and trace why the same URL is processed more than once. Done means a URL is crawled only once during a crawl run, with a regression test covering duplicate URLs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100