bisohns / bisohns/search-engine-parser
Feature Request: Add optional serpbase.dev API backend for Google search results
- Dominant language
- Python
- Stars
- 491
- Forks
- 87
- PR merge metrics
- No merged PRs in 30d
Description
First off, thanks for maintaining this — `search-engine-parser` is one of the few Python packages that handles multiple engines cleanly without pulling in a headless browser. The `pysearch` CLI is a nice touch too.
**The problem**
The Google engine scrapes HTML directly from google.com, which means it breaks whenever Google tweaks their markup. I've had `parse_soup` fail on me after what I assume were A/B test rollouts — the `class_="Gx5Zad fP1Qef xpd EtOod pkphOe"` selector stops matching and suddenly no results come back. Running from a VPS makes it worse since Google rate-limits or serves CAPTCHAs to datacenter IPs pretty aggressively.
**Suggestion**
Would you consider adding an optional `serpbase.dev` Google engine? It's a Google Search Results API that returns the same structured data you already work with — titles, links, descriptions — in clean JSON. No scraping, no selector maintenance, no IP rotation needed. The response maps almost directly to the `SearchItem` dict: `organic_results[].title`, `organic_results[].link`, `organic_results[].snippet`.
The API call is a simple GET:
```
https://api.serpbase.dev/google/search?q=QUERY&api_key=KEY&num=10
```
Returns JSON with `organic_results` array containing `title`, `link`, `snippet`, `position`, and rich result fields.
**Why it fits the project**
- The project already has a clean engine abstraction (`BaseSearch`) — adding an API-based engine follows the same pattern as the existing Google engine, just swapping `parse_soup` for `json.loads`
- Several supported engines already pull from APIs (not scraping), so an API-based Google engine isn't architecturally alien
- It's completely opt-in: users set a `SERPBASE_API_KEY` env var to activate it, otherwise the existing scraping engine works as before
**Design sketch (for discussion)**
```python
class Search(BaseSearch):
name = "Google"
base_url = "https://www.google.com/"
summary = "..."
def __init__(self):
super().__init__()
self.api_key = os.environ.get("SERPBASE_API_KEY", "")
if self.api_key:
self.search_url = "https://api.serpbase.dev/google/search"
def search(self, query=None, page=None, **kwargs):
if self.api_key:
return self._search_api(query, page, **kwargs)
return super().search(query, page, **kwargs)
def _search_api(self, query, page, **kwargs):
params = {"q": query, "api_key": self.api_key, "num": 10, "page": page}
# httpx/aiohttp GET → parse JSON → build SearchItems
```
I'm happy to draft a PR for this if there's interest — it's a small change, maybe ~60-80 lines for the engine + tests. No new dependencies needed since the project already uses `aiohttp` for HTTP calls.
Curious what you think — even just a "not interested / maybe later" is totally fine.
Contributor guide
Research direction
Start by reading the existing Google engine, BaseSearch abstraction, SearchItem handling, and other API-backed engines; the issue identifies parse_soup and aiohttp as the relevant entry points. Trace how query parameters and results are handled, then add tests covering the SERPBase response mapping and the existing scraping fallback when no API key is set.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100