deezer / deezer/spleeter

[BUG] Why Training on musdb16hq halts or does not complete

Open
#732 6 comments 0 reactions 0 assignees View on GitHub
bug invalid
Dominant language
Python
Stars
28.4k
Forks
3.1k
PR merge metrics
No merged PRs in 30d

Description

- [x] I didn't find a similar issue already open.
- [x] I read the documentation (README AND Wiki)
- [x] I have installed FFMpeg
- [x] My problem is related to Spleeter only, not a derivative product (such as Webapplication, or GUI provided by others)

## Description

## Step to reproduce

1. Installed using `.pip install python..`
2. Run as `administrator...`
3. Got `.NO..` error

## Output

```bash
After I run ...spleeter train -p 4stems-finetune.json -d E:\musdb18hq
I never get the training complete msg.
```

## Environment

| | |
| ----------------- | ------------------------------- |
| OS | Windows10|
| Installation type | pip |
| RAM available | 16Gb|
| Hardware spec | GTX 760 GPU / CPU Intel(R) Core(TM) i5 CPU 750 @ 2.67GHz 2.66 GHz|

## Additional context
I ran training on musdb_config.json and 4stems-finetune.json many times repeatedly and its starts as expected then gets about midway or 3/4 of the ways then just hangs for hours doing nothing. I also tried using only half of the musdb dataset and did the same thing. After 1 week of trying to complete this task, with multiple downloads of the musdb dataset, it still keeps failing. All my datasets contains 100 train / 50 test files. I ran the evaluation on the incompleted training results and my table is way off, I'm not sure what's going on and why its halting midway or 3/4 of the way. Am I missing files?..or corrupt files?..incompatabilities?...I'm running python 3.8.10 and spleeter 2.3.0
Any help or suggestions would be appreciated.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing `spleeter train -p 4stems-finetune.json -d E:\musdb18hq` with the reported Python 3.8.10, Spleeter 2.3.0, and Windows 10 environment. Compare the behavior with `musdb_config.json`, the reduced dataset, and the stated hardware; done means identifying a reproducible cause for the mid-training hang or documenting the missing information needed to isolate it.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.