deepspeedai / deepspeedai/DeepSpeedExamples

Inquiry about Embedding Concatenation in DeepSpeed-VisualChat

Open
#800 4 comments 1 reaction 1 assignee View on GitHub

@xiaoxiawu-microsoft is already working on this.

Since Nov 14, 2023.

Dominant language
Python
Stars
6.8k
Forks
1.1k
Avg merge
2d 16h
Merged PRs (30d)
1

Description

@yaozhewei First, I'd like to extend my gratitude for the incredible work you've been doing with DeepSpeedExamples. It's truly commendable and has been a great resource for the community.

As I was exploring the code, particularly the section where language and visual embeddings are concatenated, I came across something that prompted a question. I noticed in this line of code:

https://github.com/microsoft/DeepSpeedExamples/blob/60e412eaa7275212e240f31055fc8b814ebe653f/applications/DeepSpeed-VisualChat/utils/model/modeling_dsvl.py#L226

that img_pos_list is reversed during the insertion of the image embedding. However, it appears that cur_img is not reversed in the process. Could there potentially be a mismatch between the visual and text information due to this? I'm curious if this is intentional for alignment purposes or if it might be an oversight.

I would appreciate any clarification you can provide on this matter.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.