cloudfour / cloudfour/lighthouse-parade

add option to limit number of pages crawled?

Open
#71 8 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
373
Forks
15
Avg merge
13h 36m
Merged PRs (30d)
34

Description

Hey this is a cool tool. Here's a feature idea assuming you're open to it.

Currently one can use:
```
--max-crawl-depth 2
```

to get, say, index page + 1st linked pages.

But maybe there are a lot of linked pages, and you just need a representative sample which is more than 1 page but not tons of pages.

So maybe another option like:

```
--max-crawled-pages N # or just --max-pages ?
```

and the crawler stops after it exhausts all pages allowed by other options, or N, whichever is smaller.

One might then use it like:

```
lighthouse-parade --max-crawl-depth 2 --max-crawled-pages 20 example.com
```

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing --max-crawl-depth option and the crawler loop that applies its limits. Add a page-count option whose value is the smaller of the requested limit and the pages allowed by other options, then verify that crawling stops at that limit.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
cli
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.