optional browser-rendered fetch for JS-heavy pages in remote ingest?
- Dominant language
- Python
- Stars
- 18.3k
- Forks
- 2.1k
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 29
Description
the remote ingest path (crawler_loader / remote loaders) fetches over http, so url sources that render their content with js come back mostly empty. would an optional browser-backed loader be useful for those, one that renders the page before extraction? it'd be a python playwright wrapper over a patched firefox with a realistic fingerprint, opt-in so the default http path stays unchanged. happy to prototype a loader against the remote parser if there's interest, otherwise no worries.
Contributor guide
Research direction
Start by reading the remote ingest path, especially crawler_loader, the remote loaders, and the remote parser mentioned in the issue. Check how HTTP fetching and extraction currently work, then determine whether an opt-in browser-backed loader is feasible. The issue does not define acceptance criteria for the prototype or its rendering and fingerprinting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- playwright, python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100