Update single job scripts like utils/data/get_utt2dur.sh, steps/compute_cmvn_stats.sh to use multiple jobs
- Dominant language
- Shell
- Stars
- 15.5k
- Forks
- 5.4k
- PR merge metrics
- No merged PRs in 30d
Description
In the large data recipes in Kaldi many single job scripts in steps/\* and utils/*, which were very fast for normal sized LVCSR tasks, consume several hours.
e.g. on fisher_swbd/aspire the `utils/data/get_utt2dur.sh` script runs for a few hours.
We need to update these scripts to add parallelization support.
Required skill: shell scripting
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the named scripts, utils/data/get_utt2dur.sh and steps/compute_cmvn_stats.sh, with the large data recipes described in the issue. Trace how each script processes its inputs and identify what must change to support multiple jobs. Done means the affected single-job scripts can use parallelization without breaking normal-sized LVCSR workflows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- shell
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100