[Feature Proposal] Native Web-to-Markdown tool with up to 98% token reduction via CleanMD L2 Engine
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 58.8k
- Forks
- 8.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 109
Description
Feature Area
Agent capabilities
Is your feature request related to a an existing bug? Please link it here.
Currently, when agents scrape web pages using standard fetch or headless browsers, the resulting raw HTML consumes massive amounts of tokens (often 50k-100k+ per page), hitting context limits and drastically increasing LLM costs. Furthermore, raw HTML exposes agents to indirect prompt injections and SSRF attacks.
This feature would allow developers to drop scraping token usage by up to 98%.
Empirical Benchmarks:
- MDN Docs: 38,308 ➡️ 1,421 tokens (-96.29%) | 82ms latency
- Ethereum Org: 95,792 ➡️ 1,904 tokens (-98.01%) | 159ms latency
Describe the solution you'd like
Implement native support for CleanMD L2 Engine as an integrated LangChain Tool. It acts as a Semantic Edge Sanitizer that parses sites deterministically and returns clean Markdown.
Crucially, it supports a Bot-to-Agent (B2A) atomic micropayment system on Base L2, enabling autonomous agents to pay-per-use (0.0001 ETH) on-chain rather than relying on developer SaaS API keys.
It can be integrated via the Model Context Protocol (MCP) using the public NPM package:
npx -y cleanmd-l2-mcp-server-release
Alternatively, the HTTP API can be invoked directly at the Edge Gateway (https://cleanmd-l2-gateway.cassiusmarcelo-cabezas.workers.dev/sanitize).
Is the core team interested in reviewing a PR for this integration? I'd be happy to contribute the connector.
Describe alternatives you've considered
Using standard BeautifulSoup, Cheerio, or headless browsers (Puppeteer/Playwright) to extract HTML. However, these solutions still pass significant DOM noise, inline styles, and JavaScript to the LLM, burning context window unnecessarily and exposing the agent to SSRF vulnerabilities.
Additional context
CleanMD L2 Engine is already published on NPM (cleanmd-l2-mcp-server-release) and provides an OpenAPI spec for easy integration. I've already tested the protocol locally and verified up to a 98% reduction in token usage for heavy sites like MDN Docs and Ethereum Org.
Willingness to Contribute
Yes, I'd be happy to submit a pull request
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No repository files or tests are named. Start by reviewing the existing tool integration and MCP entry points, then evaluate the cleanmd-l2-mcp-server-release package and the listed sanitize endpoint, including SSRF, external-network, and payment requirements. Done would require an agreed integration design, implementation, and tests for the supported behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- nodejs, openapi, python
- Domain
- ai, api, blockchain, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100