deezer / deezer/spleeter

[Discussion] Uncertainties In Evaluation

Open
#654 0 comments 1 reaction 0 assignees View on GitHub
question
Dominant language
Python
Stars
28.4k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

I tried evaluating the metrics on musdb18hq test set with the pretrained [4stems](https://github.com/deezer/spleeter/releases/tag/v1.4.0) model.
But the result is not as good as the one mentioned in [wiki page](https://github.com/deezer/spleeter/wiki/Separation-Performances#pretrained-models)
I'm wondering what are the uncertainties existed in the evaluation process which results in a different performance?

**Here are the results:**

INFO:spleeter:musdb evaluation done
INFO:spleeter:vocals:
INFO:spleeter:SDR: 6.354
INFO:spleeter:SAR: 5.740
INFO:spleeter:SIR: 13.628
INFO:spleeter:ISR: 12.989
INFO:spleeter:drums:
INFO:spleeter:SDR: 5.687
INFO:spleeter:SAR: 4.855
INFO:spleeter:SIR: 12.559
INFO:spleeter:ISR: 10.579
INFO:spleeter:bass:
INFO:spleeter:SDR: 4.959
INFO:spleeter:SAR: 3.440
INFO:spleeter:SIR: 10.419
INFO:spleeter:ISR: 6.071
INFO:spleeter:other:
INFO:spleeter:SDR: 4.225
INFO:spleeter:SAR: 3.612
INFO:spleeter:SIR: 6.038
INFO:spleeter:ISR: 10.474

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.