BuilderIO / BuilderIO/gpt-crawler
Request to support PDF scraping
- Dominant language
- TypeScript
- Stars
- 22.4k
- Forks
- 2.4k
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
Thank you for this amazing repo. I am trying to use this on a website which also has 100s of pdfs. The crawler is unable to get the content from the PDFs. It fails with the error:
PlaywrightCrawler: Request failed and reached maximum retries. page.goto: net::ERR_ABORTED
It will be great if request for crawling through PDFs can be added as well.
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue identifies PlaywrightCrawler and the page.goto ERR_ABORTED failure as the starting points. Trace how PlaywrightCrawler handles PDF requests, then determine how PDF content should be retrieved and exposed; done means PDF URLs crawl successfully and their content is available to the generated knowledge files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100