instadeepai / instadeepai/marl-eval

[Feature] User-Selected Normalisation Values

Open
#65 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
87
Forks
8
PR merge metrics
No merged PRs in 30d

Description

# Description
Hey! A super useful feature would be allowing users to set the values that are used for normalisation of tasks. For example then one could easily do PPO-Normalisation or Human-Performance-Normalisation etc. Additionally, one could simply set the true max and min values of a task. This is really a much needed feature since the way normalisation is done here can actually cause serious undesired effects. Through the addition of new algorithms and more data thereby changing the normalisation, you can actually change the outcome severely of previous experiments. For example, if you ran PPO and SAC and compared performance, you might see that one is statistically significant in its improvement, however as soon as you add a new algorithm this changes the normalisation and can completely change this result. Addiyionally, if you are making two separate plots of different subsets of algorithms, these scores can be normalised completely differently leading to misleading results when looking across the graphs.

# Desired Outcome
I imagine the best way would simply be to allow the user to input a dictionary mapping env suite and task to min and max score.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.