Add Circle/x402 paid crawl endpoint for autonomous agents

Proposal: add an optional Circle/x402 paid crawl endpoint for autonomous agents

I ran a static Circle/x402 readiness scan against Crawl4AI to see whether it could become an agent-payable service. Crawl4AI is a strong fit because agents already need on-demand crawling, extraction, and LLM-ready web data.

Readiness summary

• Score from the scan: 46/100 — good candidate, medium effort.
• The repo has meaningful primitives for crawling jobs, schemas, rate/concurrency concerns, and receipt/proof-like result concepts.
• I did not find x402 or Circle Agent Wallet/Marketplace-specific documentation.
• The scanner did not fully detect Crawl4AI's Python/FastAPI-style routes, but source references show Docker/API surfaces such as /crawl, /crawl/job, /crawl/stream, and /llm/job patterns.

Why this is useful

AI agents often need one-off crawling or extraction without a long-lived API key, dashboard account, or shared credit pool. A Circle/x402 paid route would let an agent:

1. Discover a paid crawl endpoint.
2. Inspect price and input schema.
3. Pay in USDC from a Circle Agent Wallet.
4. Receive crawl output plus a receipt/proof object.

Suggested first implementation

Start with a narrow additive endpoint rather than changing the whole auth or deployment model:

POST /x402/crawl
  -> return 402 Payment Required when no valid payment is supplied
  -> verify x402 payment on Base/USDC
  -> run the existing crawl/job pipeline
  -> return { data, receipt }

A receipt shape for agent buyers could look like:

{
  "receipt": {
    "amount": "0.01",
    "asset": "USDC",
    "network": "base",
    "resource": "POST /x402/crawl",
    "requestHash": "sha256:...",
    "resultHash": "sha256:...",
    "jobId": "crawl_..."
  }
}

Circle Agent Marketplace readiness docs

If this is interesting, I would also document:

• supported network(s)
• USDC price per crawl or per page
• request JSON schema
• response JSON schema
• receipt/proof object
• timeout and retry behavior
• how Circle Agent Wallet / x402 clients should pay

I generated this from Budcle, a small scanner for identifying Circle/x402 adoption gaps in open-source API projects: https://github.com/batikanor/Budcle

This is intentionally additive: existing Crawl4AI library usage, Docker deployment, and auth choices can remain unchanged while adding one agent-payable path for x402-capable clients.

Start by locating the existing /crawl, /crawl/job, /crawl/stream, and /llm/job API entry points and the crawl/job pipeline they use. Review how request schemas, authentication, and results are handled before defining the additive /x402/crawl behavior. Done would include the paid route, payment verification, crawl output with a receipt, and documentation of supported networks, pricing, schemas, and retries.