google-research / google-research/composed_image_retrieval
About training detail of text encoder
- Dominant language
- Shell
- Stars
- 197
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
Hi, thanks for your contribution. I have one question about the training details.
In the **Figure2.left** of the paper, both of visual encoder and text encoder are **frozen** during training,meaning that no gradients will be generated.
But in the **get_loss_img2text** function of **src/trainer.py**, it seems that some gradients will be generated in **get_text_features** function.
Is there something I'm misunderstanding? I would be grateful if you could answer my question.
```
def get_text_features(model, token_features, args):
text = tokenize("a photo of")
text = text.cuda(args.gpu, non_blocking=True)
text = text.view(1, -1)
text = text.repeat(token_features.size(0), 1)
text_features = model.encode_text_img(text, token_features)
return text_features
def get_loss_img2text(model, img2text, images, loss_img, loss_txt, args, memory=None):
with torch.no_grad():
image_features = model.encode_image(images)
token_features = img2text(image_features)
text_features = get_text_features(model, token_features, args)
```
Contributor guide
Research direction
Start by reading src/trainer.py, especially get_loss_img2text and get_text_features, alongside the paper's Figure 2 description. Confirm how gradients and frozen encoders are handled in this path. Done means adding a clear explanation to the issue or project documentation that resolves the apparent discrepancy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- machine-learning
- Issue type
- Documentation
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100