EveryVoiceTTS / EveryVoiceTTS/EveryVoice
Add a way to set the batch size at inference time
- Dominant language
- Python
- Stars
- 45
- Forks
- 4
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 14
Description
### Description & Motivation
Before #697, at inference time we hard coded a batch size of 1.
With #697 we now hard code a batch size of 16 to do multiple chunks at once.
Ideally, we want to let the user control the batch size at inference time.
### Pitch
- [ ] Choose a default that is generally suitable for the kind of hardware used at inference time. Is that really 16? Or should we restrict it to 8 or even 4?
- [ ] Add a command-line argument to let the user decide. This is probably best added to `apply_inference_shared_parameters` and the associated `_interface` function in `everyvoice/base_cli/interfaces.py` and `.../helpers.py`.
### Alternatives
Maybe we could use the config's `training.batch_size` as a default value?
### Additional context
At training time, we can expect significant computing resources to be available, but inference might get run on smaller computing devices. While using the config's batch size might make intuitive sense for a default, it may not be suitable at inference time. Whatever we set here as the default should be a value we expect to be typically suitable at inference time.
Contributor guide
Assessment
This issue has not been assessed yet.