bisohns / bisohns/search-engine-parser

Feature Request: Add optional serpbase.dev API backend for Google search results

Open
#190 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
491
Forks
87
PR merge metrics
No merged PRs in 30d

Description

First off, thanks for maintaining this — `search-engine-parser` is one of the few Python packages that handles multiple engines cleanly without pulling in a headless browser. The `pysearch` CLI is a nice touch too.

**The problem**

The Google engine scrapes HTML directly from google.com, which means it breaks whenever Google tweaks their markup. I've had `parse_soup` fail on me after what I assume were A/B test rollouts — the `class_="Gx5Zad fP1Qef xpd EtOod pkphOe"` selector stops matching and suddenly no results come back. Running from a VPS makes it worse since Google rate-limits or serves CAPTCHAs to datacenter IPs pretty aggressively.

**Suggestion**

Would you consider adding an optional `serpbase.dev` Google engine? It's a Google Search Results API that returns the same structured data you already work with — titles, links, descriptions — in clean JSON. No scraping, no selector maintenance, no IP rotation needed. The response maps almost directly to the `SearchItem` dict: `organic_results[].title`, `organic_results[].link`, `organic_results[].snippet`.

The API call is a simple GET:
```
https://api.serpbase.dev/google/search?q=QUERY&api_key=KEY&num=10
```

Returns JSON with `organic_results` array containing `title`, `link`, `snippet`, `position`, and rich result fields.

**Why it fits the project**

- The project already has a clean engine abstraction (`BaseSearch`) — adding an API-based engine follows the same pattern as the existing Google engine, just swapping `parse_soup` for `json.loads`
- Several supported engines already pull from APIs (not scraping), so an API-based Google engine isn't architecturally alien
- It's completely opt-in: users set a `SERPBASE_API_KEY` env var to activate it, otherwise the existing scraping engine works as before

**Design sketch (for discussion)**

```python
class Search(BaseSearch):
name = "Google"
base_url = "https://www.google.com/"
summary = "..."

def __init__(self):
super().__init__()
self.api_key = os.environ.get("SERPBASE_API_KEY", "")
if self.api_key:
self.search_url = "https://api.serpbase.dev/google/search"

def search(self, query=None, page=None, **kwargs):
if self.api_key:
return self._search_api(query, page, **kwargs)
return super().search(query, page, **kwargs)

def _search_api(self, query, page, **kwargs):
params = {"q": query, "api_key": self.api_key, "num": 10, "page": page}
# httpx/aiohttp GET → parse JSON → build SearchItems
```

I'm happy to draft a PR for this if there's interest — it's a small change, maybe ~60-80 lines for the engine + tests. No new dependencies needed since the project already uses `aiohttp` for HTTP calls.

Curious what you think — even just a "not interested / maybe later" is totally fine.

Contributor guide

Open the contributing guide

Research direction

Start by reading the existing Google engine, BaseSearch abstraction, SearchItem handling, and other API-backed engines; the issue identifies parse_soup and aiohttp as the relevant entry points. Trace how query parameters and results are handled, then add tests covering the SERPBase response mapping and the existing scraping fallback when no API key is set.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.