opensearch-project / opensearch-project/data-prepper
website crawler - "source"
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
This is a new feature request. We'd like to have Data Prepper have web site crawling capabilities that can crawl web sites and facilitate the ingestion of web pages into OpenSearch.
Describe the solution you'd like
Introduce a "webcrawler source" that would provide ability to crawl a public website on a periodic basis (on-demand or a schedule), respecting the configuration of the website and rate-limiting of the requests, filtering (including/excluding pages), etc... On pages that were acquired, the ability to store content in OpenSearch for search and discovery.
Describe alternatives you've considered (Optional)
Use of a Selenium web crawler and a Chromium driver, then filtering, enriching the content, and storing it in OpenSearch.
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named in the issue. Start by defining the webcrawler source scope, including scheduling, robots/configuration compliance, rate limiting, page filtering, and OpenSearch ingestion; done requires an agreed design and acceptance criteria for these capabilities.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100