crewAIInc / crewAIInc/crewAI

[FEATURE] A web-scraping tool whose browser session survives between runs

Open
#7,380 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

vendor-pitch
Dominant language
Python
Stars
58.8k
Forks
8.5k
Avg merge
1d 15h
Merged PRs (30d)
109

Description

Feature Area

Integration with external tools

Is your feature request related to a an existing bug? Please link it here.

NA

Describe the solution you'd like

Every browser tool under web-scraping today starts from nothing on each call: a fresh browser, a stock automation fingerprint, and whatever IP the process has. That is fine for a public page and useless behind a login, because the session dies with the browser, so a crew that has to read a signed-in page cannot keep its state between steps.

I would like a tool whose unit of state is a browser profile rather than a session: cookies, storage, a fingerprint and a proxy that all persist, so the second call opens the page already signed in, and one browser is reused across the calls of a single crew run.

Concretely, a AntibrowLoadTool(profile="research-01") that takes a url (and an optional CSS selector), returns the page's visible text truncated to a limit, and exposes close() for cleanup.

Describe alternatives you've considered
  • SeleniumScrapingTool with a user-data-dir: keeps cookies, but the fingerprint is a stock automation build and the setup is left to the user.
  • The hosted-browser tools (BrowserbaseLoadTool, HyperbrowserLoadTool): sessions are remote and per-call, and they bill by browser time.
  • Doing it outside the crew and passing text in: works, but then the login step is not something an agent can drive.
Additional Context

I am affiliated with AntiBrow, which is why I can implement it: a PR is ready with the tool, 9 unit tests, the optional extra, and the docs page in en/ar/ko/pt-BR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the existing browser tools under web-scraping and the proposed AntibrowLoadTool(profile="research-01") interface. Then inspect the nine unit tests, optional extra, and documentation pages in en/ar/ko/pt-BR mentioned in the issue. Done means the profile state and browser reuse behavior work across calls, cleanup is exposed, and the documented interface is covered.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
tooling, web-dev
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.