BuilderIO / BuilderIO/gpt-crawler

Adds for autoScroll for crawling the multi pages?

Open
#30 3 comments 4 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
22.4k
Forks
2.4k
PR merge metrics
No merged PRs in 30d

Description

I just worked for our platform pages with origin code and that couldn't provide me full information on pages.

Therefore, i added autoScroll code in main.ts for this and it worked perfectly.
(I think it is better than increasing the numbers of waitForSelectorTimeout.)

```CLI
async function autoScroll(page: Page) {
await page.evaluate(async () => {
await new Promise((resolve, reject) => {
var totalHeight = 0;
var distance = 100;
var timer = setInterval(() => {
var scrollHeight = document.body.scrollHeight;
window.scrollBy(0, distance);
totalHeight += distance;

if (totalHeight >= scrollHeight) {
clearInterval(timer);
resolve();
}
}, 100);
});
});
}

if (process.env.NO_CRAWL !== "true") {
const crawler = new PlaywrightCrawler({
async requestHandler({ request, page, enqueueLinks, log, pushData }) {
try {
if (config.cookie) {
const cookie = {
name: config.cookie.name,
value: config.cookie.value,
url: request.loadedUrl,
};
await page.context().addCookies([cookie]);
}

const title = await page.title();
log.info(`Crawling ${request.loadedUrl}...`);

await page.waitForSelector(config.selector, {
timeout: config.waitForSelectorTimeout,
});

await autoScroll(page);

const html = await getPageHtml(page);
await pushData({ title, url: request.loadedUrl, html });

if (config.onVisitPage) {
await config.onVisitPage({ page, pushData });
}

await enqueueLinks({
globs: [config.match],
});
} catch (error) {
log.error(`Error crawling ${request.loadedUrl}: ${error}`);
}
},
maxRequestsPerCrawl: config.maxPagesToCrawl,
// headless: false,
});

await crawler.run([config.url]);
}
```

If you think this is good enough for crawling, hope this will be helpful for other users.

Thank you for your work btw!

I really appreciate for that!

Thank you.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in main.ts at the PlaywrightCrawler requestHandler, especially the waitForSelector and getPageHtml calls. Review the proposed autoScroll flow and verify it against crawling pages that load content while scrolling. Done means the crawler captures the additional page content without relying only on a larger waitForSelectorTimeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, typescript
Domain
backend, web-dev
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.