BuilderIO / BuilderIO/gpt-crawler
Adds for autoScroll for crawling the multi pages?
- Dominant language
- TypeScript
- Stars
- 22.4k
- Forks
- 2.4k
- PR merge metrics
- No merged PRs in 30d
Description
I just worked for our platform pages with origin code and that couldn't provide me full information on pages.
Therefore, i added autoScroll code in main.ts for this and it worked perfectly.
(I think it is better than increasing the numbers of waitForSelectorTimeout.)
```CLI
async function autoScroll(page: Page) {
await page.evaluate(async () => {
await new Promise((resolve, reject) => {
var totalHeight = 0;
var distance = 100;
var timer = setInterval(() => {
var scrollHeight = document.body.scrollHeight;
window.scrollBy(0, distance);
totalHeight += distance;
if (totalHeight >= scrollHeight) {
clearInterval(timer);
resolve();
}
}, 100);
});
});
}
if (process.env.NO_CRAWL !== "true") {
const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, enqueueLinks, log, pushData }) {
try {
if (config.cookie) {
const cookie = {
name: config.cookie.name,
value: config.cookie.value,
url: request.loadedUrl,
};
await page.context().addCookies([cookie]);
}
const title = await page.title();
log.info(`Crawling ${request.loadedUrl}...`);
await page.waitForSelector(config.selector, {
timeout: config.waitForSelectorTimeout,
});
await autoScroll(page);
const html = await getPageHtml(page);
await pushData({ title, url: request.loadedUrl, html });
if (config.onVisitPage) {
await config.onVisitPage({ page, pushData });
}
await enqueueLinks({
globs: [config.match],
});
} catch (error) {
log.error(`Error crawling ${request.loadedUrl}: ${error}`);
}
},
maxRequestsPerCrawl: config.maxPagesToCrawl,
// headless: false,
});
await crawler.run([config.url]);
}
```
If you think this is good enough for crawling, hope this will be helpful for other users.
Thank you for your work btw!
I really appreciate for that!
Thank you.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in main.ts at the PlaywrightCrawler requestHandler, especially the waitForSelector and getPageHtml calls. Review the proposed autoScroll flow and verify it against crawling pages that load content while scrolling. Done means the crawler captures the additional page content without relying only on a larger waitForSelectorTimeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, typescript
- Domain
- backend, web-dev
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100