Cannot Reproduce MusDB Evaluation Results
- Dominant language
- Python
- Stars
- 28.4k
- Forks
- 3.1k
- PR merge metrics
- No merged PRs in 30d
Description
- [✅] I didn't find a similar issue already open.
- [✅] I read the documentation (README AND Wiki)
- [✅] I have installed FFMpeg
- [✅] My problem is related to Spleeter only, not a derivative product (such as Webapplication, or GUI provided by others)
## Description
Despite multiple training runs using the provided MusDB configuration, I cannot reproduce the results shown on the MusDB evaluation table. Training/evaluation is performed using the provided musdb_config.json file. My experiment results in ~10% less average SDR performance (Avg. SDR of 4.059 vs expected 4.43, Spleeter team's result). Full result comparision can be found under output section.
## Step to reproduce
1. Installed using pip (Python 3.8 venv, Spleeter 2.2.2, Tensorflow-GPU 2.6.0, Nvidia RTX 2080Ti, CUDA 11.4, CuDNN Toolkit 11.2)
2. Run as (in virtual env):
- spleeter train -p configs/musdb_config.json -d /path/to/musdb18 --verbose
- check train/validation csvs and audio data loaded
- spleeter evaluate -p configs/musdb_config.json --musd_dir /path/to/musdb18 -o /some_output_eval_path/ --verbose
3. Evaluation result for avg. SDR is ~10% less than Spleeter's MusDB evaluation.
## Output
**My musdb_config.json Result**
- Vocals SDR: 4.668
- Bass SDR: 3.889
- Drums SDR: 4.496
- Other SDR: 3.181
- Average SDR: 4.059
**Expected: Spleeter Team's musdb_config.json Result**
- Vocals SDR: 5.10
- Bass SDR: 4.27
- Drums SDR: 5.15
- Other SDR: 3.21
- Average SDR: 4.43
## Environment
| | |
| ----------------- | ------------------------------- |
| OS | Linux 18.04 |
| Installation type | pip |
| RAM available | 64GB |
| Hardware spec | Nvidia RTX 2080Ti, Intel i7-8700k |
## Additional context
Tools: Python 3.8 venv, Spleeter 2.2.2, Tensorflow-GPU 2.6.0, Nvidia RTX 2080Ti, CUDA 11.4, CuDNN Toolkit 11.2
Contributor guide
Research direction
Start by reproducing the training and evaluation commands with configs/musdb_config.json, checking the listed dataset, training/validation CSVs, audio data, and environment versions. Compare the resulting SDR values with the expected MusDB table results; done means the discrepancy is explained and the documented configuration produces or clearly accounts for the reported metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, tensorflow
- Domain
- audio-video-rtc, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100