andreagrandi / andreagrandi/draftomen
Train and benchmark a historical-pick ranker
- 主要语言
- Python
- 星标
- 0
- 派生
- 0
- 平均合并
- 1 小时 23 分钟
- 30 天内合并 PR
- 181
描述
## Problem
Draftgoblin has a documented ML design and an existing DG Score benchmark, but no trained historical-pick model or evidence that ML improves recommendations. Production integration must remain gated until an offline model beats the current heuristic on held-out data.
## Proposed change
Consume the dataset produced by the pick-ranking dataset workflow and add a reproducible, research-only LightGBM LambdaRank experiment. Compare its rankings with raw 17Lands win rate and the current DG Score on identical held-out picks.
Produce aggregate reports and an artifact manifest, but do not integrate the model into live, replay, TUI, or backtest recommendation paths in this ticket.
## Acceptance Criteria
- [ ] Keep LightGBM and supporting ML packages isolated from Draftgoblin's runtime dependencies.
- [ ] Train `LGBMRanker` with the `lambdarank` objective, pick-level groups, a fixed seed, validation-based early stopping, and recorded parameters.
- [ ] Compare raw 17L WR, current DG Score, and ML rankings on exactly the same eligible held-out picks.
- [ ] Report top-1/top-3/top-5 agreement, mean actual-pick rank, MRR, and NDCG at 1, 3, and 5.
- [ ] Break results down by set, format, and `open`, `building`, and `locked` phases; report all-draft and trophy/outcome strata separately.
- [ ] Calculate paired 95% confidence intervals using bootstrap samples clustered by remapped draft ID.
- [ ] Evaluate at least two set/format datasets and report skipped or unresolved rows for each model.
- [ ] Measure local single-pack prediction latency against Draftgoblin's 1.5-second display budget.
- [ ] Record model version, feature schema, set/format, source metadata, split ranges, parameters, metrics, checksum, and compatibility information in an artifact manifest.
- [ ] State whether the promotion gate passed: positive MRR improvement over DG Score with a 95% interval excluding zero on two datasets, and no phase losing more than one percentage point of top-3 agreement.
- [ ] Keep DG Score as the default regardless of the result; runtime integration requires a separate follow-up issue.
- [ ] Add deterministic fixture tests and run the complete training and evaluation workflow against locally supplied public data.
贡献指南
这个仓库没有索引到贡献指南
评估
这个 Issue 还没有评估数据。