andreagrandi / andreagrandi/draftomen

Train and benchmark a historical-pick ranker

未关闭
#88 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
draftgoblin enhancement size: L
主要语言
Python
星标
0
派生
0
平均合并
1 小时 23 分钟
30 天内合并 PR
181

描述

## Problem

Draftgoblin has a documented ML design and an existing DG Score benchmark, but no trained historical-pick model or evidence that ML improves recommendations. Production integration must remain gated until an offline model beats the current heuristic on held-out data.

## Proposed change

Consume the dataset produced by the pick-ranking dataset workflow and add a reproducible, research-only LightGBM LambdaRank experiment. Compare its rankings with raw 17Lands win rate and the current DG Score on identical held-out picks.

Produce aggregate reports and an artifact manifest, but do not integrate the model into live, replay, TUI, or backtest recommendation paths in this ticket.

## Acceptance Criteria

- [ ] Keep LightGBM and supporting ML packages isolated from Draftgoblin's runtime dependencies.
- [ ] Train `LGBMRanker` with the `lambdarank` objective, pick-level groups, a fixed seed, validation-based early stopping, and recorded parameters.
- [ ] Compare raw 17L WR, current DG Score, and ML rankings on exactly the same eligible held-out picks.
- [ ] Report top-1/top-3/top-5 agreement, mean actual-pick rank, MRR, and NDCG at 1, 3, and 5.
- [ ] Break results down by set, format, and `open`, `building`, and `locked` phases; report all-draft and trophy/outcome strata separately.
- [ ] Calculate paired 95% confidence intervals using bootstrap samples clustered by remapped draft ID.
- [ ] Evaluate at least two set/format datasets and report skipped or unresolved rows for each model.
- [ ] Measure local single-pack prediction latency against Draftgoblin's 1.5-second display budget.
- [ ] Record model version, feature schema, set/format, source metadata, split ranges, parameters, metrics, checksum, and compatibility information in an artifact manifest.
- [ ] State whether the promotion gate passed: positive MRR improvement over DG Score with a 95% interval excluding zero on two datasets, and no phase losing more than one percentage point of top-3 agreement.
- [ ] Keep DG Score as the default regardless of the result; runtime integration requires a separate follow-up issue.
- [ ] Add deterministic fixture tests and run the complete training and evaluation workflow against locally supplied public data.

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。