Refactor: Migrate search engine metadata from JSON to class attributes (LLM provider pattern)

Problem

Currently, search engines define their metadata (description, strengths, weaknesses, requirements, etc.) in verbose JSON configuration files. This creates several issues:

1. Duplication: Static metadata that never changes is stored in configuration files
2. Boilerplate: Each new search engine requires ~200-300 lines of JSON for metadata
3. Maintenance: Metadata is separated from the implementation, making it harder to keep in sync
4. Discoverability: Developers need to look in two places (class + JSON) to understand a search engine

Solution

Adopt the LLM provider pattern where static metadata is defined as class attributes:

Current Pattern (Search Engines)

class MySearchEngine(BaseSearchEngine):
    is_public = True
    is_scientific = True
    # ... implementation

// settings_my_search.json (~250 lines)
{
  "search.engine.web.my_search.description": { ... },
  "search.engine.web.my_search.strengths": { ... },
  "search.engine.web.my_search.weaknesses": { ... },
  "search.engine.web.my_search.requires_api_key": { ... },
  // ... many more entries
}

New Pattern (LLM Providers - proposed for search engines)

class MySearchEngine(BaseSearchEngine):
    # Static metadata as class attributes
    engine_name = "My Search"
    engine_key = "MY_SEARCH"
    description = "Search for..."
    
    # Behavioral flags
    is_public = True
    is_scientific = True
    
    # Requirements
    requires_api_key = True
    requires_llm = True
    
    # Quality metrics
    reliability = 0.90
    
    # Metadata
    strengths = ["feature 1", "feature 2", ...]
    weaknesses = ["limitation 1", "limitation 2", ...]
    
    # ... implementation

// settings_my_search.json (~50 lines - only runtime config)
{
  "search.engine.web.my_search.api_key": { ... },
  "search.engine.web.my_search.max_results": { ... },
  // Only user-specific settings
}

Benefits

1. Self-documenting: All metadata is in one place with the implementation
2. Less boilerplate: Reduce JSON config from ~250 lines to ~50 lines
3. Better IDE support: Class attributes have autocomplete and type hints
4. Consistency: Same pattern as LLM providers (already in codebase)
5. Easier maintenance: Update metadata where the code is
6. Cleaner architecture: Separation of static metadata vs runtime configuration

Reference Implementation

✅ EBSCO search engine has been implemented using the new pattern:

• File: src/local_deep_research/web_search_engines/engines/search_engine_ebsco.py
• Shows all metadata as class attributes (lines 31-87)
• Minimal JSON config with only credentials and runtime settings

Search Engines to Migrate

• Brave Search (search_engine_brave.py)
• Guardian (search_engine_guardian.py)
• Tavily (search_engine_tavily.py)
• PubMed (search_engine_pubmed.py)
• Semantic Scholar (search_engine_semantic_scholar.py)
• Perplexity (search_engine_perplexity.py)
• SearXNG (search_engine_searxng.py)
• Google Custom Search (search_engine_google.py)
• Bing (search_engine_bing.py)
• DuckDuckGo (search_engine_duckduckgo.py)
• Any other search engines in engines/ directory

Migration Steps (per engine)

For each search engine:

1. Add class attributes for metadata:

   engine_name = "Engine Name"
   engine_key = "ENGINE_KEY"
   description = "..."
   requires_api_key = True/False
   requires_llm = True/False
   reliability = 0.0-1.0
   strengths = [...]
   weaknesses = [...]
   

2. Update JSON config to remove duplicated metadata:

   • Remove description, strengths, weaknesses entries
   • Remove reliability, requires_api_key, requires_llm entries
   • Keep only module_path, class_name, and runtime settings

3. Test that the engine still works correctly

4. Update documentation if needed

Optional: Auto-Discovery System

Similar to LLM providers, we could create an auto-discovery system that:

• Scans search engine classes
• Extracts metadata from class attributes
• Generates EngineInfo objects
• Provides structured access to metadata

This would further reduce boilerplate, but is optional for this refactor.

Priority

Medium - This is a quality-of-life improvement that will make the codebase cleaner and easier to maintain. It's not urgent but should be done before adding many more search engines.

Related

• Issue #1066 (EBSCO API Support) - implemented using the new pattern
• LLM provider implementation in src/local_deep_research/llm/providers/