deezer / deezer/spleeter

Cannot Reproduce MusDB Evaluation Results

Open
#670 2 comments 0 reactions 0 assignees View on GitHub
bug invalid
Dominant language
Python
Stars
28.4k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

- [✅] I didn't find a similar issue already open.
- [✅] I read the documentation (README AND Wiki)
- [✅] I have installed FFMpeg
- [✅] My problem is related to Spleeter only, not a derivative product (such as Webapplication, or GUI provided by others)

## Description

Despite multiple training runs using the provided MusDB configuration, I cannot reproduce the results shown on the MusDB evaluation table. Training/evaluation is performed using the provided musdb_config.json file. My experiment results in ~10% less average SDR performance (Avg. SDR of 4.059 vs expected 4.43, Spleeter team's result). Full result comparision can be found under output section.

## Step to reproduce

1. Installed using pip (Python 3.8 venv, Spleeter 2.2.2, Tensorflow-GPU 2.6.0, Nvidia RTX 2080Ti, CUDA 11.4, CuDNN Toolkit 11.2)
2. Run as (in virtual env):
- spleeter train -p configs/musdb_config.json -d /path/to/musdb18 --verbose
- check train/validation csvs and audio data loaded
- spleeter evaluate -p configs/musdb_config.json --musd_dir /path/to/musdb18 -o /some_output_eval_path/ --verbose

3. Evaluation result for avg. SDR is ~10% less than Spleeter's MusDB evaluation.

## Output

**My musdb_config.json Result**
- Vocals SDR: 4.668
- Bass SDR: 3.889
- Drums SDR: 4.496
- Other SDR: 3.181
- Average SDR: 4.059

**Expected: Spleeter Team's musdb_config.json Result**
- Vocals SDR: 5.10
- Bass SDR: 4.27
- Drums SDR: 5.15
- Other SDR: 3.21
- Average SDR: 4.43

## Environment

| | |
| ----------------- | ------------------------------- |
| OS | Linux 18.04 |
| Installation type | pip |
| RAM available | 64GB |
| Hardware spec | Nvidia RTX 2080Ti, Intel i7-8700k |

## Additional context

Tools: Python 3.8 venv, Spleeter 2.2.2, Tensorflow-GPU 2.6.0, Nvidia RTX 2080Ti, CUDA 11.4, CuDNN Toolkit 11.2

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the training and evaluation commands with configs/musdb_config.json, checking the listed dataset, training/validation CSVs, audio data, and environment versions. Compare the resulting SDR values with the expected MusDB table results; done means the discrepancy is explained and the documented configuration produces or clearly accounts for the reported metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, tensorflow
Domain
audio-video-rtc, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.