arc53 / arc53/DocsGPT

optional browser-rendered fetch for JS-heavy pages in remote ingest?

Open
#2,576 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
18.3k
Forks
2.1k
Avg merge
1d 2h
Merged PRs (30d)
29

Description

the remote ingest path (crawler_loader / remote loaders) fetches over http, so url sources that render their content with js come back mostly empty. would an optional browser-backed loader be useful for those, one that renders the page before extraction? it'd be a python playwright wrapper over a patched firefox with a realistic fingerprint, opt-in so the default http path stays unchanged. happy to prototype a loader against the remote parser if there's interest, otherwise no worries.

Contributor guide

Open the contributing guide

Research direction

Start by reading the remote ingest path, especially crawler_loader, the remote loaders, and the remote parser mentioned in the issue. Check how HTTP fetching and extraction currently work, then determine whether an opt-in browser-backed loader is feasible. The issue does not define acceptance criteria for the prototype or its rendering and fingerprinting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
playwright, python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.