ringo380 / ringo380/QueryGrade

Retrain ML model on real feedback (replace synthetic bootstrap)

Open
#92 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area-ml-system blocked-on-real-usage priority-medium type-ml-improvement
Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Context

The currently-ACTIVE model (query_grader, version 20260521_021306) was trained as a bootstrap on synthetic/seed TrainingData to give the issue-#5 monitoring pipeline an ACTIVE target. Its metrics reflect that:

  • Training accuracy: 0.804
  • Validation accuracy: 0.712 (just over the 0.7 auto-deploy threshold)
  • Test accuracy: 0.095 — effectively no real predictive power.

It satisfies "an ACTIVE model exists so monitoring evaluates," but it is not a quality predictor. The low-confidence / low-user-agreement alerts it generates reflect exactly this.

Goal

Replace the synthetic bootstrap with a model trained on accumulated real user feedback once enough has been collected.

Tasks

  • Define a minimum real-feedback sample threshold before retraining (current ML_MIN_TRAINING_SAMPLES=50 counts synthetic seed rows — may want a separate "validated real feedback" gate).
  • Confirm process_ml_feedbackTrainingData flow is converting real QueryFeedback into training samples.
  • Establish a retrain cadence (the ML_AUTO_RETRAIN / beat machinery exists) and a quality gate higher than the bootstrap's 0.7 val / 0.095 test.
  • Re-evaluate model_type: bootstrap is QUERY_GRADER; the grading path expects HYBRID_SCORER (per CLAUDE.md) once HybridQueryGrader is wired in.

Notes

  • Training runs in-container: railway ssh -s querygrade-worker "python manage.py train_ml_model …" (local railway run can't reach postgres.railway.internal).
  • Related: model-artifact durability (#91) becomes relevant once predictions actually load the model.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing process_ml_feedback into TrainingData, then inspect ML_MIN_TRAINING_SAMPLES, ML_AUTO_RETRAIN, the beat machinery, and the train_ml_model management command. Read CLAUDE.md and the HybridQueryGrader path to resolve the model_type requirement. Done means real feedback is gated and used for retraining, cadence and quality thresholds are established, and the resulting model improves meaningfully beyond the bootstrap metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
postgres, python
Domain
backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.