Better Trueskill evaluation

Continuing the thread from #43 here because #43 is closed.

@kachayev mentioned

> Note sure this is the most convenient avenue for the discussion (let me know if you want to move this to Discord).

I am ok with both. Github issues are archived and easy to view for future users, which is nice. Whereas Discord is easier to do quick chat, so both have pros and cons.

> I finally got time to play around with this proposal. I think it's a good idea! Playing games with least predictable outcome should reduce overall uncertainty about skill estimates. Initially I though this is used for matchmaking for training but it seems it's only used for evaluation. Empirically the same approach would be problematic when deciding on opponents for training but just for measurements.

Yes, currently it's only used for evaluation: the Trueskill of the reference agent are fixed and we only update the trueskill of the training agent.

> There's still an interesting concern, or even a topic for research/investigations. TrueSkill relies on the prior estimate for draws. For some games, even when following random policy, there is a little chance to get draw (e.g. Chess, or Go). But in other games, it actually takes more sophisticated policy to avoid "drawing" all the time. Specifically keeping in mind practical limitation of using "timestep budget" that typically results in draw. Working with MicroRTS environment we get around this (partially) because we have scripted bots that are quite good from get go. Other options would be an SL policy (imitation learning) to bootstrap the initial population where the estimate of TrueSkill would have reasonably low sigma (and reasonably low prob of drawing).

> Questions now are:
* how sensitive algorithm to the quality of prior?
* how to properly estimate draws prior for a given population?
* how does it change with training?
* and more

These questions are a bit outside of my realm, haha, but I think are interesting questions worth investigating.

> Another question: do we need to re-evaluate skill for existing players when injecting new players into the population. Empirically for large populations, the answer is "yes". It would be interesting to run experiments here for a small size population and only a few newcomers. As we talk about latent variable estimate, we don't ground truth to compare to. But we can run simplified version with only a few match ups selected for each new player vs. full round robin. And measure how different the final result is.

The short answer is no at least in this paradigm. I am following OpenAI Five and AlphaStar's evaluation of fixing the trueskill for reference agents.

See