websearch remains unreliable when DDG and Bing HTML are blocked; fallback diagnostics hide attempts

Summary

The built-in websearch tool still has no reliable usable backend when DuckDuckGo and Bing HTML both reject automated requests. On an affected macOS aarch64 host, ordinary documentation searches fail consistently even after the fallback added for #270.

This is both a backend reliability gap and a diagnostics bug: when DuckDuckGo errors and the Bing fallback returns an empty result set, execute preserves the earlier DuckDuckGo error and reports only that error. The user cannot tell that Bing was attempted or why it produced no results.

Reproduction

Environment:

• jcode v0.81.14-dev (167648aca)
• macOS aarch64
• no Bing API key or SearXNG URL configured

Run five ordinary websearch requests using engine = "duckduckgo", for example:

site:docs.github.com GitHub Actions concurrency

Observed for 5/5 DuckDuckGo requests:

DuckDuckGo served an anti-bot challenge page (anomaly challenge) instead of results.
This is commonly caused by TLS fingerprinting or IP reputation on Linux.
Falling back to another engine if configured.

Run the same query with engine = "bing".

Observed:

No results found for: site:docs.github.com GitHub Actions concurrency

Direct webfetch of known GitHub Docs URLs works from the same session, so general outbound HTTP is healthy.

Code path

• crates/jcode-app-core/src/tool/websearch.rs:87-123 builds the primary/fallback list.
• execute stores only last_error. An Ok(empty) fallback does not replace or annotate it.
• crates/jcode-app-core/src/tool/websearch.rs:179-225 detects DuckDuckGo's challenge.
• Bing HTML may return an empty/block page that is not classified as an error.
• The only API-shaped escape hatch that does not scrape a public search page is user-configured SearXNG. Public instances are not a dependable default, and self-hosting is too much setup for a core agent tool.

Expected behavior

1. websearch should have at least one supported API-backed provider path suitable for reliable agent use.
2. Fallback results should report every attempted engine and outcome, for example:
   • DuckDuckGo: anti-bot challenge
   • Bing HTML: empty/block page
   • SearXNG: not configured
3. Configured API providers should be preferred over HTML scraping. Paid or metered providers must only be used when explicitly configured.
4. A doctor/status path should show which search backends are configured and usable before an agent spends calls on them.
5. HTML engines should remain best-effort fallbacks, not the reliability foundation.

Suggested architecture

Introduce a small SearchBackend result contract with structured outcomes such as Results, Empty, Blocked, Unconfigured, and Failed. Resolve a configured route once, then try explicit fallbacks while collecting outcomes. Render an aggregate failure with actionable setup guidance.

For the reliable backend, evaluate one API-native provider such as Brave Search, Tavily, or Exa, plus the existing self-hosted SearXNG route. Keep credentials provider-specific and opt-in. Avoid silently routing queries through arbitrary public SearXNG instances.

Acceptance criteria

• Fixture test: DuckDuckGo blocked plus Bing empty reports both outcomes.
• Fixture test: configured API backend succeeds without touching HTML engines.
• Fixture test: no configured reliable backend returns one actionable aggregate error.
• Integration test or doctor check validates backend configuration without issuing a billable search.
• Tool description/config docs distinguish reliable API routes from best-effort HTML scraping.

Related: #270 fixed the silent empty-result symptom and added SearXNG, but did not provide a dependable default/backend or aggregate fallback diagnostics.

Start in crates/jcode-app-core/src/tool/websearch.rs, especially lines 87-123 and 179-225, and trace execute's error handling and existing fallback routes. Use the stated fixture and doctor/configuration acceptance criteria to define coverage for blocked, empty, successful, and unconfigured backends. Done means attempted engines and outcomes are reported, configured API routes are preferred, and documentation distinguishes reliable routes from HTML fallbacks.