andreagrandi / andreagrandi/draftomen

Train and benchmark a historical-pick ranker

オープン
#88 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
draftgoblin enhancement size: L
主要言語
Python
スター
0
フォーク
0
平均マージ
1時間 23分
マージ済み PR(30日)
181

説明

## Problem

Draftgoblin has a documented ML design and an existing DG Score benchmark, but no trained historical-pick model or evidence that ML improves recommendations. Production integration must remain gated until an offline model beats the current heuristic on held-out data.

## Proposed change

Consume the dataset produced by the pick-ranking dataset workflow and add a reproducible, research-only LightGBM LambdaRank experiment. Compare its rankings with raw 17Lands win rate and the current DG Score on identical held-out picks.

Produce aggregate reports and an artifact manifest, but do not integrate the model into live, replay, TUI, or backtest recommendation paths in this ticket.

## Acceptance Criteria

- [ ] Keep LightGBM and supporting ML packages isolated from Draftgoblin's runtime dependencies.
- [ ] Train `LGBMRanker` with the `lambdarank` objective, pick-level groups, a fixed seed, validation-based early stopping, and recorded parameters.
- [ ] Compare raw 17L WR, current DG Score, and ML rankings on exactly the same eligible held-out picks.
- [ ] Report top-1/top-3/top-5 agreement, mean actual-pick rank, MRR, and NDCG at 1, 3, and 5.
- [ ] Break results down by set, format, and `open`, `building`, and `locked` phases; report all-draft and trophy/outcome strata separately.
- [ ] Calculate paired 95% confidence intervals using bootstrap samples clustered by remapped draft ID.
- [ ] Evaluate at least two set/format datasets and report skipped or unresolved rows for each model.
- [ ] Measure local single-pack prediction latency against Draftgoblin's 1.5-second display budget.
- [ ] Record model version, feature schema, set/format, source metadata, split ranges, parameters, metrics, checksum, and compatibility information in an artifact manifest.
- [ ] State whether the promotion gate passed: positive MRR improvement over DG Score with a 95% interval excluding zero on two datasets, and no phase losing more than one percentage point of top-3 agreement.
- [ ] Keep DG Score as the default regardless of the result; runtime integration requires a separate follow-up issue.
- [ ] Add deterministic fixture tests and run the complete training and evaluation workflow against locally supplied public data.

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。