EveryVoiceTTS / EveryVoiceTTS/EveryVoice

Add a way to set the batch size at inference time

Open
#701 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Python
Stars
45
Forks
4
Avg merge
1d 2h
Merged PRs (30d)
14

Description

### Description & Motivation

Before #697, at inference time we hard coded a batch size of 1.
With #697 we now hard code a batch size of 16 to do multiple chunks at once.

Ideally, we want to let the user control the batch size at inference time.

### Pitch

- [ ] Choose a default that is generally suitable for the kind of hardware used at inference time. Is that really 16? Or should we restrict it to 8 or even 4?
- [ ] Add a command-line argument to let the user decide. This is probably best added to `apply_inference_shared_parameters` and the associated `_interface` function in `everyvoice/base_cli/interfaces.py` and `.../helpers.py`.

### Alternatives

Maybe we could use the config's `training.batch_size` as a default value?

### Additional context

At training time, we can expect significant computing resources to be available, but inference might get run on smaller computing devices. While using the config's batch size might make intuitive sense for a default, it may not be suitable at inference time. Whatever we set here as the default should be a value we expect to be typically suitable at inference time.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.