acl-org / acl-org/reviewer-paper-matching

Track-wise Optimization Options

オープン
#28 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
26
フォーク
4
PR マージ指標
30日以内にマージされた PR はありません

説明

Some observations from @fxiafxia and others who worked with the reviewer assignment system for ACL-2021:
- Breaking down the optimization problem into subproblems/submatrices for each track _greatly_ improved the overall runtime of the optimization
- This makes sense because it eliminates a huge amount of unused parameters
- We implemented this optimization with our acl-2021 branch
- With the subproblem optimization, the step that takes by far the longest now is the aggregation of paper-paper similarity scores into paper-reviewer similarity scores
- The results of the optimization system seem to be very "unbalanced". I.e. a few reviewers will get the max number of papers and lots of others will get none or the minimum
- It is hard to "tune" the min and max number of papers to assign, since the needs of each track are quite different from each other. Some tracks have an excess of reviewers, some have barely enough

Based on these observations, we had a few thoughts for the future of the system:
- Would it be possible to only compute similarity scores and aggregate scores within tracks, since this step seems to take a lot of time globally? (if the `--track` parameter is specified)
- Could you read in a configuration file that has the min and max papers per reviewer on a track-by-track basis, and defaulting to the global setting when this is not included?
- Could you somehow tweak the objective function to consider "balanced" assignment?
- A naive formulation of this might be adding up the differences between each reviewer's assigned number of papers and the "track average", if you divided the papers equally among the reviewers of the track
- Others have put forward ideas taking this notion of "balance" further, e.g. balancing inexperienced vs experienced reviewers, academia vs industry, geographic balance, etc

コントリビューションガイド

このリポジトリのコントリビューションガイドは索引されていません

調査の方向性

No files or tests are named. Start by reviewing the ACL-2021 branch's track-wise optimization and the paper-paper to paper-reviewer score aggregation. The issue proposes several possible directions—track-scoped scores, per-track limits, and balanced assignments—but does not define which one to implement or what completion criteria should be.

索引モデルが issue の本文から書いたものです。

評価

技術スタック
python
領域
data
issue の種類
機能追加
難易度
5/5
見積もり時間
1週間以上
活発さ
停滞
明瞭さ
説明が足りない
初心者へのやさしさ
25/100

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。