gonczor / gonczor/DIYHelper

Improve knowledge retrieval with Hackaday taxonomy and prefix matching

Open
#5 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
0
Forks
0
Avg merge
1h 45m
Merged PRs (30d)
3

Description

## Problem

Knowledge retrieval uses exact PostgreSQL full-text lexemes from article titles and content. A broad query such as `ATmega` may not find an article about `ATmega328P`, even though the article is relevant. Hackaday categories (the “Posted in” section) and tags are currently discarded during ingestion, losing useful classification metadata.

## Proposed scope

- Extract Hackaday categories from the “Posted in” section.
- Extract Hackaday article tags.
- Persist categories and tags with each knowledge article.
- Include categories and tags in the PostgreSQL search vector with a higher weight than body content.
- Add safe prefix matching so a broader term such as `ATmega` can retrieve `ATmega328P`, while retaining stronger ranking for exact matches.
- Include taxonomy metadata in stored ingestion artifacts and model reference formatting where useful.
- Define a re-ingestion or backfill path for articles that already exist without taxonomy metadata.
- OR-only matching for normalized query lexemes.
- I want info about Atmega328P retrieving atmega328p articles.
- Articles matching both atmega328 and esp32 ranking above single-product matches.
- Single-product matches remaining eligible.
- Acceptance of generic-term candidate pollution.
- No curated filler-word list in scope.

## Acceptance criteria

- An imported article retains its Hackaday categories and tags.
- Searching by a stored category or tag returns the corresponding article.
- Searching for `ATmega` can return an otherwise relevant `ATmega328P` article.
- Exact title and tag matches rank above weaker body or prefix matches.
- Source filtering, candidate limits, article limits, and token budgets continue to apply.
- Updating taxonomy invalidates any cached token count when the rendered reference changes.
- Parser, repository, ranking, migration, and ingestion behavior are covered by tests.
- Tests for OR matching, ranking, filler words, empty lexemes, limits, and source filtering.

## Out of scope

- Conversation follow-up handling and reuse of previously gathered knowledge.
- Embedding/vector search.
- A general-purpose manually maintained classification ontology.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the ingestion parser, article persistence, PostgreSQL search vector construction, ranking, and cached reference rendering; the issue names these entry points but no files or tests. Define the taxonomy storage and backfill path before implementing, then cover parser, repository, ranking, migration, ingestion, OR matching, limits, source filtering, and cache invalidation as specified.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgresql, python
Domain
backend, databases, search
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.