crewAIInc / crewAIInc/crewAI

[Feature Proposal] Native Web-to-Markdown tool with up to 98% token reduction via CleanMD L2 Engine

Open
#7,579 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature-request vendor-pitch
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Feature Area

Agent capabilities

Is your feature request related to a an existing bug? Please link it here.

Currently, when agents scrape web pages using standard fetch or headless browsers, the resulting raw HTML consumes massive amounts of tokens (often 50k-100k+ per page), hitting context limits and drastically increasing LLM costs. Furthermore, raw HTML exposes agents to indirect prompt injections and SSRF attacks.

This feature would allow developers to drop scraping token usage by up to 98%.
Empirical Benchmarks:

  • MDN Docs: 38,308 ➡️ 1,421 tokens (-96.29%) | 82ms latency
  • Ethereum Org: 95,792 ➡️ 1,904 tokens (-98.01%) | 159ms latency
Describe the solution you'd like

Implement native support for CleanMD L2 Engine as an integrated LangChain Tool. It acts as a Semantic Edge Sanitizer that parses sites deterministically and returns clean Markdown.

Crucially, it supports a Bot-to-Agent (B2A) atomic micropayment system on Base L2, enabling autonomous agents to pay-per-use (0.0001 ETH) on-chain rather than relying on developer SaaS API keys.

It can be integrated via the Model Context Protocol (MCP) using the public NPM package:
npx -y cleanmd-l2-mcp-server-release

Alternatively, the HTTP API can be invoked directly at the Edge Gateway (https://cleanmd-l2-gateway.cassiusmarcelo-cabezas.workers.dev/sanitize).

Is the core team interested in reviewing a PR for this integration? I'd be happy to contribute the connector.

Describe alternatives you've considered

Using standard BeautifulSoup, Cheerio, or headless browsers (Puppeteer/Playwright) to extract HTML. However, these solutions still pass significant DOM noise, inline styles, and JavaScript to the LLM, burning context window unnecessarily and exposing the agent to SSRF vulnerabilities.

Additional context

CleanMD L2 Engine is already published on NPM (cleanmd-l2-mcp-server-release) and provides an OpenAPI spec for easy integration. I've already tested the protocol locally and verified up to a 98% reduction in token usage for heavy sites like MDN Docs and Ethereum Org.

Willingness to Contribute

Yes, I'd be happy to submit a pull request

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No repository files or tests are named. Start by reviewing the existing tool integration and MCP entry points, then evaluate the cleanmd-l2-mcp-server-release package and the listed sanitize endpoint, including SSRF, external-network, and payment requirements. Done would require an agreed integration design, implementation, and tests for the supported behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
nodejs, openapi, python
Domain
ai, api, blockchain, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.