andreagrandi / andreagrandi/draftomen

Train and benchmark a historical-pick ranker

Aperta
#88 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
draftgoblin enhancement size: L
Lingua principale
Python
Stelle
0
Fork
0
Merge medio
1h 23m
PR unite (30g)
181

Descrizione

## Problem

Draftgoblin has a documented ML design and an existing DG Score benchmark, but no trained historical-pick model or evidence that ML improves recommendations. Production integration must remain gated until an offline model beats the current heuristic on held-out data.

## Proposed change

Consume the dataset produced by the pick-ranking dataset workflow and add a reproducible, research-only LightGBM LambdaRank experiment. Compare its rankings with raw 17Lands win rate and the current DG Score on identical held-out picks.

Produce aggregate reports and an artifact manifest, but do not integrate the model into live, replay, TUI, or backtest recommendation paths in this ticket.

## Acceptance Criteria

- [ ] Keep LightGBM and supporting ML packages isolated from Draftgoblin's runtime dependencies.
- [ ] Train `LGBMRanker` with the `lambdarank` objective, pick-level groups, a fixed seed, validation-based early stopping, and recorded parameters.
- [ ] Compare raw 17L WR, current DG Score, and ML rankings on exactly the same eligible held-out picks.
- [ ] Report top-1/top-3/top-5 agreement, mean actual-pick rank, MRR, and NDCG at 1, 3, and 5.
- [ ] Break results down by set, format, and `open`, `building`, and `locked` phases; report all-draft and trophy/outcome strata separately.
- [ ] Calculate paired 95% confidence intervals using bootstrap samples clustered by remapped draft ID.
- [ ] Evaluate at least two set/format datasets and report skipped or unresolved rows for each model.
- [ ] Measure local single-pack prediction latency against Draftgoblin's 1.5-second display budget.
- [ ] Record model version, feature schema, set/format, source metadata, split ranges, parameters, metrics, checksum, and compatibility information in an artifact manifest.
- [ ] State whether the promotion gate passed: positive MRR improvement over DG Score with a 95% interval excluding zero on two datasets, and no phase losing more than one percentage point of top-3 agreement.
- [ ] Keep DG Score as the default regardless of the result; runtime integration requires a separate follow-up issue.
- [ ] Add deterministic fixture tests and run the complete training and evaluation workflow against locally supplied public data.

Guida per i contributori

Nessuna guida per i contributori indicizzata per questo repository

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.