[Discussion] Uncertainties In Evaluation
- Dominant language
- Python
- Stars
- 28.4k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
I tried evaluating the metrics on musdb18hq test set with the pretrained [4stems](https://github.com/deezer/spleeter/releases/tag/v1.4.0) model.
But the result is not as good as the one mentioned in [wiki page](https://github.com/deezer/spleeter/wiki/Separation-Performances#pretrained-models)
I'm wondering what are the uncertainties existed in the evaluation process which results in a different performance?
**Here are the results:**
INFO:spleeter:musdb evaluation done
INFO:spleeter:vocals:
INFO:spleeter:SDR: 6.354
INFO:spleeter:SAR: 5.740
INFO:spleeter:SIR: 13.628
INFO:spleeter:ISR: 12.989
INFO:spleeter:drums:
INFO:spleeter:SDR: 5.687
INFO:spleeter:SAR: 4.855
INFO:spleeter:SIR: 12.559
INFO:spleeter:ISR: 10.579
INFO:spleeter:bass:
INFO:spleeter:SDR: 4.959
INFO:spleeter:SAR: 3.440
INFO:spleeter:SIR: 10.419
INFO:spleeter:ISR: 6.071
INFO:spleeter:other:
INFO:spleeter:SDR: 4.225
INFO:spleeter:SAR: 3.612
INFO:spleeter:SIR: 6.038
INFO:spleeter:ISR: 10.474
Contributor guide
Assessment
This issue has not been assessed yet.