kaldi-asr / kaldi-asr/kaldi

Update single job scripts like utils/data/get_utt2dur.sh, steps/compute_cmvn_stats.sh to use multiple jobs

Open
#984 6 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted stale
Dominant language
Shell
Stars
15.5k
Forks
5.4k
PR merge metrics
No merged PRs in 30d

Description

In the large data recipes in Kaldi many single job scripts in steps/\* and utils/*, which were very fast for normal sized LVCSR tasks, consume several hours.
e.g. on fisher_swbd/aspire the `utils/data/get_utt2dur.sh` script runs for a few hours.

We need to update these scripts to add parallelization support.

Required skill: shell scripting

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by comparing the named scripts, utils/data/get_utt2dur.sh and steps/compute_cmvn_stats.sh, with the large data recipes described in the issue. Trace how each script processes its inputs and identify what must change to support multiple jobs. Done means the affected single-job scripts can use parallelization without breaking normal-sized LVCSR workflows.

Written by the indexing model from the issue text.

Assessment

Tech stack
shell
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.