facebookresearch / facebookresearch/fairscale

Error Freezing Weights

Open
#1,126 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
3.4k
Forks
293
PR merge metrics
No merged PRs in 30d

Description

I am trying to freeze weights of linear layers but I get this error:
```
module.weight.requires_grad = not freeze
RuntimeError: you can only change requires_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn't require differentiation use var_no_grad = var.detach().
```

and in my case the `module` is `ColumnParallelLinear` whose weight does seem to be a leaf variable:
https://github.com/facebookresearch/fairscale/blob/164cc0f3170b4a3951dd84dda29c3e1504ac4d6e/fairscale/nn/model_parallel/layers.py#L262

so I am not sure why I am getting this error message.

Contributor guide

Open the contributing guide

Research direction

Start in fairscale/nn/model_parallel/layers.py around line 262 and reproduce the reported requires_grad assignment with ColumnParallelLinear. Trace how its weight is created and determine why it is not treated as a leaf variable; done means the cause and the supported freezing behavior are established, with a regression test if the repository provides a suitable test location.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.