instadeepai / instadeepai/marl-eval

Hard-coding in confidence interval calculations for single tasks

Open
#66 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
87
Forks
8
PR merge metrics
No merged PRs in 30d

Description

In [this](https://github.com/instadeepai/marl-eval/blob/main/marl_eval/utils/data_processing_utils.py#L143) line:

- Sample size (i.e., number of seeds) is hard-coded to ten. This is problematic for any evaluation that is not ten seeds because the resulting confidence intervals will be incorrect; and
- The critical value for the 95% confidence interval is hard-coded to use the normal distribution (1.96), which is incorrect for small sample sizes like $n=10$. It is generally recommended to use the t-distribution for $n\le30$ and an unknown population standard deviation (which is the case in RL, since we are estimating the standard deviation from our sample). For example, the two-sided 95% critical value for $n=10$ using Student's t-distribution is 2.262, not 1.96. Student's t-distribution will make the confidence intervals more conservative for small sample sizes.

Contributor guide

Open the contributing guide

Research direction

Start at marl_eval/utils/data_processing_utils.py line 143 and trace how the sample size and confidence interval are calculated from evaluation seeds. Verify the behavior for seed counts other than ten and for small samples using the appropriate critical value; done means the resulting 95% intervals no longer depend on hard-coded values.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.