Add a feeze option in Wav2Vec2 and HuBERT bundles
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.9k
- Forks
- 799
- Avg merge
- 58m
- Merged PRs (30d)
- 3
Description
🚀 The feature
In some research cases, the Wav2Vec2 or HuBERT is expected to be frozen (i.e. make reuqires_grad=False for all params).
- Users use it as a feature extractor that generates pretty robust representations and jointly train with down-stream neural networks. In this case, people don't want to fine-tune the Wav2Vec2/HuBERT to avoid over-fitting on the downstream dataset.
- Users want to jointly train their front-end model and use the gradient of Wav2Vec2/HuBERT only for back-propagation.
It'll be good to add an argument to set the model to frozen state so that users don't need to set requires_grad=False by themselves.
Motivation, pitch
SpeechBrain has similar implementation by adding freeze argument when initilizing the model. https://github.com/speechbrain/speechbrain/blob/f1f421b3bb58dabc75d67c3fd5f6e3359943b927/speechbrain/lobes/models/fairseq_wav2vec.py#L78
if self.freeze:
self.model.eval()
# Freeze parameters
for param in model.parameters():
param.requires_grad = False
cc @mthrok
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Wav2Vec2 and HuBERT bundle entry points and reviewing the linked SpeechBrain freeze implementation. Check the existing bundle tests, if present, and define completion as an option that freezes the relevant model parameters and supports the stated feature-extraction and downstream-training uses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100