pytorch / pytorch/audio

Add a feeze option in Wav2Vec2 and HuBERT bundles

Open
#1,995 15 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

improvement module: pipelines
Dominant language
Python
Stars
2.9k
Forks
799
Avg merge
58m
Merged PRs (30d)
3

Description

🚀 The feature

In some research cases, the Wav2Vec2 or HuBERT is expected to be frozen (i.e. make reuqires_grad=False for all params).

  • Users use it as a feature extractor that generates pretty robust representations and jointly train with down-stream neural networks. In this case, people don't want to fine-tune the Wav2Vec2/HuBERT to avoid over-fitting on the downstream dataset.
  • Users want to jointly train their front-end model and use the gradient of Wav2Vec2/HuBERT only for back-propagation.

It'll be good to add an argument to set the model to frozen state so that users don't need to set requires_grad=False by themselves.

Motivation, pitch

SpeechBrain has similar implementation by adding freeze argument when initilizing the model. https://github.com/speechbrain/speechbrain/blob/f1f421b3bb58dabc75d67c3fd5f6e3359943b927/speechbrain/lobes/models/fairseq_wav2vec.py#L78

  if self.freeze:
      self.model.eval()
      # Freeze parameters
      for param in model.parameters():
          param.requires_grad = False

cc @mthrok

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the Wav2Vec2 and HuBERT bundle entry points and reviewing the linked SpeechBrain freeze implementation. Check the existing bundle tests, if present, and define completion as an option that freezes the relevant model parameters and supports the stated feature-extraction and downstream-training uses.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.