ringo380 / ringo380/QueryGrade
Retrain ML model on real feedback (replace synthetic bootstrap)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Context
The currently-ACTIVE model (query_grader, version 20260521_021306) was trained as a bootstrap on synthetic/seed TrainingData to give the issue-#5 monitoring pipeline an ACTIVE target. Its metrics reflect that:
- Training accuracy: 0.804
- Validation accuracy: 0.712 (just over the 0.7 auto-deploy threshold)
- Test accuracy: 0.095 — effectively no real predictive power.
It satisfies "an ACTIVE model exists so monitoring evaluates," but it is not a quality predictor. The low-confidence / low-user-agreement alerts it generates reflect exactly this.
Goal
Replace the synthetic bootstrap with a model trained on accumulated real user feedback once enough has been collected.
Tasks
- Define a minimum real-feedback sample threshold before retraining (current
ML_MIN_TRAINING_SAMPLES=50counts synthetic seed rows — may want a separate "validated real feedback" gate). - Confirm
process_ml_feedback→TrainingDataflow is converting realQueryFeedbackinto training samples. - Establish a retrain cadence (the
ML_AUTO_RETRAIN/ beat machinery exists) and a quality gate higher than the bootstrap's 0.7 val / 0.095 test. - Re-evaluate model_type: bootstrap is
QUERY_GRADER; the grading path expectsHYBRID_SCORER(per CLAUDE.md) onceHybridQueryGraderis wired in.
Notes
- Training runs in-container:
railway ssh -s querygrade-worker "python manage.py train_ml_model …"(localrailway runcan't reachpostgres.railway.internal). - Related: model-artifact durability (#91) becomes relevant once predictions actually load the model.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing process_ml_feedback into TrainingData, then inspect ML_MIN_TRAINING_SAMPLES, ML_AUTO_RETRAIN, the beat machinery, and the train_ml_model management command. Read CLAUDE.md and the HybridQueryGrader path to resolve the model_type requirement. Done means real feedback is gated and used for retraining, cadence and quality thresholds are established, and the resulting model improves meaningfully beyond the bootstrap metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgres, python
- Domain
- backend, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100