andreagrandi / andreagrandi/draftomen

Train and benchmark a historical-pick ranker

Ouverte
#88 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
draftgoblin enhancement size: L
Langage dominant
Python
Étoiles
0
Forks
0
Merge moyen
1 h 23 min
PR mergées (30 j)
181

Description

## Problem

Draftgoblin has a documented ML design and an existing DG Score benchmark, but no trained historical-pick model or evidence that ML improves recommendations. Production integration must remain gated until an offline model beats the current heuristic on held-out data.

## Proposed change

Consume the dataset produced by the pick-ranking dataset workflow and add a reproducible, research-only LightGBM LambdaRank experiment. Compare its rankings with raw 17Lands win rate and the current DG Score on identical held-out picks.

Produce aggregate reports and an artifact manifest, but do not integrate the model into live, replay, TUI, or backtest recommendation paths in this ticket.

## Acceptance Criteria

- [ ] Keep LightGBM and supporting ML packages isolated from Draftgoblin's runtime dependencies.
- [ ] Train `LGBMRanker` with the `lambdarank` objective, pick-level groups, a fixed seed, validation-based early stopping, and recorded parameters.
- [ ] Compare raw 17L WR, current DG Score, and ML rankings on exactly the same eligible held-out picks.
- [ ] Report top-1/top-3/top-5 agreement, mean actual-pick rank, MRR, and NDCG at 1, 3, and 5.
- [ ] Break results down by set, format, and `open`, `building`, and `locked` phases; report all-draft and trophy/outcome strata separately.
- [ ] Calculate paired 95% confidence intervals using bootstrap samples clustered by remapped draft ID.
- [ ] Evaluate at least two set/format datasets and report skipped or unresolved rows for each model.
- [ ] Measure local single-pack prediction latency against Draftgoblin's 1.5-second display budget.
- [ ] Record model version, feature schema, set/format, source metadata, split ranges, parameters, metrics, checksum, and compatibility information in an artifact manifest.
- [ ] State whether the promotion gate passed: positive MRR improvement over DG Score with a 95% interval excluding zero on two datasets, and no phase losing more than one percentage point of top-3 agreement.
- [ ] Keep DG Score as the default regardless of the result; runtime integration requires a separate follow-up issue.
- [ ] Add deterministic fixture tests and run the complete training and evaluation workflow against locally supplied public data.

Guide de contribution

Aucun guide de contribution indexé pour ce dépôt

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.