BuilderIO / BuilderIO/gpt-crawler

Wildcard support

Open
#7 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
22.4k
Forks
2.4k
PR merge metrics
No merged PRs in 30d

Description

I've noticed you can't currently use any regex when defining urls, but is there some other way to leverage different subdomains or wildcard characters?

For example, I want to crawl multiple subdomains that follow a similar structure `https://pco[NAME].zendesk.com/`. I was thinking I could change the `match` field to accept a string array, and also couldn't use regex to wildcard the [NAME] piece of the subdomain. Is there some other way to achieve this?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by locating the URL configuration and the `match` field handling described in the issue. Determine how URL patterns are currently evaluated, then define support for matching multiple subdomains such as `https://pco[NAME].zendesk.com/`; done means the requested wildcard or equivalent matching behavior works for that example.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
web-dev
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.