lakshyakumar / lakshyakumar/Knowledge-Base-AI
📚 Knowledge Base Creation from Entire Websites
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Description:
We aim to support full-website scraping and transform the collected content into a queryable semantic knowledge base. The objective is to recursively extract, chunk, and embed website content and store it in a vector database to support future natural language queries with contextual understanding.
Acceptance Criteria:
- Crawl all reachable pages within a domain (respecting
robots.txtand rate limits). - Chunk scraped content into semantically meaningful blocks (paragraphs, sections).
- Generate embeddings using OpenAI or SentenceTransformers.
- Store content and metadata (URL, title, headings) in a vector DB (Chroma, FAISS, etc.).
- Provide API endpoints to query and fetch relevant semantic chunks.
📘 Note: Please read and follow the Contribution Guide before starting work on any of the tasks listed below.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading CONTRIBUTING.md, then inspect the existing project entry points for scraping and API behavior. Break the work into crawling, semantic chunking, embeddings, vector storage, and query endpoints; it is done when all listed acceptance criteria, including metadata and crawling limits, are supported.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- fastapi, python
- Domain
- ai, api, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100