google-research / google-research/composed_image_retrieval

About training detail of text encoder

Open
#24 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Shell
Stars
197
Forks
20
PR merge metrics
No merged PRs in 30d

Description

Hi, thanks for your contribution. I have one question about the training details.

In the **Figure2.left** of the paper, both of visual encoder and text encoder are **frozen** during training,meaning that no gradients will be generated.

But in the **get_loss_img2text** function of **src/trainer.py**, it seems that some gradients will be generated in **get_text_features** function.

Is there something I'm misunderstanding? I would be grateful if you could answer my question.
```
def get_text_features(model, token_features, args):
text = tokenize("a photo of")
text = text.cuda(args.gpu, non_blocking=True)
text = text.view(1, -1)
text = text.repeat(token_features.size(0), 1)
text_features = model.encode_text_img(text, token_features)
return text_features

def get_loss_img2text(model, img2text, images, loss_img, loss_txt, args, memory=None):
with torch.no_grad():
image_features = model.encode_image(images)
token_features = img2text(image_features)
text_features = get_text_features(model, token_features, args)
```

Contributor guide

Open the contributing guide

Research direction

Start by reading src/trainer.py, especially get_loss_img2text and get_text_features, alongside the paper's Figure 2 description. Confirm how gradients and frozen encoders are handled in this path. Done means adding a clear explanation to the issue or project documentation that resolves the apparent discrepancy.

Written by the indexing model from the issue text.

Assessment

Tech stack
python, pytorch
Domain
machine-learning
Issue type
Documentation
Difficulty
1/5
Estimated time
Under an hour
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.