Improve knowledge retrieval with Hackaday taxonomy and prefix matching
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- Avg merge
- 1h 45m
- Merged PRs (30d)
- 3
Description
## Problem
Knowledge retrieval uses exact PostgreSQL full-text lexemes from article titles and content. A broad query such as `ATmega` may not find an article about `ATmega328P`, even though the article is relevant. Hackaday categories (the “Posted in” section) and tags are currently discarded during ingestion, losing useful classification metadata.
## Proposed scope
- Extract Hackaday categories from the “Posted in” section.
- Extract Hackaday article tags.
- Persist categories and tags with each knowledge article.
- Include categories and tags in the PostgreSQL search vector with a higher weight than body content.
- Add safe prefix matching so a broader term such as `ATmega` can retrieve `ATmega328P`, while retaining stronger ranking for exact matches.
- Include taxonomy metadata in stored ingestion artifacts and model reference formatting where useful.
- Define a re-ingestion or backfill path for articles that already exist without taxonomy metadata.
- OR-only matching for normalized query lexemes.
- I want info about Atmega328P retrieving atmega328p articles.
- Articles matching both atmega328 and esp32 ranking above single-product matches.
- Single-product matches remaining eligible.
- Acceptance of generic-term candidate pollution.
- No curated filler-word list in scope.
## Acceptance criteria
- An imported article retains its Hackaday categories and tags.
- Searching by a stored category or tag returns the corresponding article.
- Searching for `ATmega` can return an otherwise relevant `ATmega328P` article.
- Exact title and tag matches rank above weaker body or prefix matches.
- Source filtering, candidate limits, article limits, and token budgets continue to apply.
- Updating taxonomy invalidates any cached token count when the rendered reference changes.
- Parser, repository, ranking, migration, and ingestion behavior are covered by tests.
- Tests for OR matching, ranking, filler words, empty lexemes, limits, and source filtering.
## Out of scope
- Conversation follow-up handling and reuse of previously gathered knowledge.
- Embedding/vector search.
- A general-purpose manually maintained classification ontology.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the ingestion parser, article persistence, PostgreSQL search vector construction, ranking, and cached reference rendering; the issue names these entry points but no files or tests. Define the taxonomy storage and backfill path before implementing, then cover parser, repository, ranking, migration, ingestion, OR matching, limits, source filtering, and cache invalidation as specified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python
- Domain
- backend, databases, search
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100