instadeepai / instadeepai/marl-eval
Hard-coding in confidence interval calculations for single tasks
- Dominant language
- Python
- Stars
- 87
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
In [this](https://github.com/instadeepai/marl-eval/blob/main/marl_eval/utils/data_processing_utils.py#L143) line:
- Sample size (i.e., number of seeds) is hard-coded to ten. This is problematic for any evaluation that is not ten seeds because the resulting confidence intervals will be incorrect; and
- The critical value for the 95% confidence interval is hard-coded to use the normal distribution (1.96), which is incorrect for small sample sizes like $n=10$. It is generally recommended to use the t-distribution for $n\le30$ and an unknown population standard deviation (which is the case in RL, since we are estimating the standard deviation from our sample). For example, the two-sided 95% critical value for $n=10$ using Student's t-distribution is 2.262, not 1.96. Student's t-distribution will make the confidence intervals more conservative for small sample sizes.
Contributor guide
Research direction
Start at marl_eval/utils/data_processing_utils.py line 143 and trace how the sample size and confidence interval are calculated from evaluation seeds. Verify the behavior for seed counts other than ten and for small samples using the appropriate critical value; done means the resulting 95% intervals no longer depend on hard-coded values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100