crewAIInc / crewAIInc/crewAI

[FEATURE] Add an AnyAPI tool for catalog-driven access to scraping and data APIs

Open
#7,253 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

vendor-pitch
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Feature Area

Integration with external tools

Is your feature request related to a an existing bug? Please link it here.

NA

Describe the solution you'd like

crewai_tools already carries a lot of single-vendor data tools - Apify, Bright Data, Exa, Firecrawl, Jina, Oxylabs, Scrapfly, SerpApi, Serper, Tavily. Each one gives an agent a fixed capability from one vendor, and adding a new data source means adding a new tool and a new key.

I would like a tool for AnyAPI, which is shaped differently: it is a gateway to hundreds of scraping and data APIs behind one key, billed per request in USD with no subscription. Because it is a catalog rather than a single endpoint, the useful shape is three tools that an agent chains:

  1. AnyApiSearchTool - ranked search over the catalog to find the endpoint that returns the data the task needs
  2. AnyApiDescribeTool - that endpoint's input and output JSON Schema and its USD price, which the agent reads before its first call
  3. AnyApiRunTool - execute the endpoint

The point of the split is that the agent picks the data source at run time instead of the developer picking it at import time. An agent asked for "recent reviews of this restaurant" searches, finds the Google Maps reviews endpoint, reads its schema, and runs it, without anyone having added a Maps-specific tool first.

Deliberately not included: any hardcoded endpoint list, per-platform convenience tools, or catalog caching. Routing, pricing and schemas belong to the gateway, so the tool stays a thin transport layer and does not need a release when the catalog changes.

Describe alternatives you've considered

A single tool taking an endpoint slug and an input. Rejected because the agent has no way to learn which slugs exist or what they accept, and every AnyAPI input schema is strict, so an invented field name fails the call. The search-then-describe-then-run split is what makes it usable without the developer knowing the catalog.

Modelling it on apify_actors_tool, which is the closest existing shape. It solves the same discovery problem for one vendor's actors; this generalises it across vendors.

Additional context

I work on AnyAPI, so this is a vendor integration and I am declaring that up front. The implementation is written, tested and ready: three BaseTool subclasses over a shared base, 14 unit tests, getanyapi declared as an optional extra the way the sibling tools declare theirs. ruff check, ruff format --check, uv lock --check and the tool-spec generator gate all pass, and I have smoke tested the search and describe paths against the live API.

Happy to be told this belongs outside the repo as a third-party package instead - I would rather ask than open an unwanted PR.

Willingness to Contribute

Yes, I'd be happy to submit a pull request

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing apify_actors_tool pattern and the three BaseTool subclasses described in the issue. Check the 14 unit tests and the optional getanyapi extra, then run ruff check, ruff format --check, uv lock --check, and the tool-spec generator; completion includes the reported live search and describe smoke tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.