lxtGH / lxtGH/OMG-Seg

GCG推理时无法批处理

Open
#52 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1.4k
Forks
55
PR merge metrics
No merged PRs in 30d

Description

作者您好,我发现在我移植的gcg推理代码中无法实现批处理,具体如下:
predict, seg_hidden_states = forward_model(
batch_inputs, pixel_values,
self.omg_tokenizer, self.omg_model, self.omg_llm,
self.omg_projector_text2vision,
self.omg_gen_config, self.omg_stop_criteria)
对于batch为4的图像输入,pixel_values包含四个元素,如下:
image
但是predict, seg_hidden_statesg尺寸不对,是对应单图的尺寸,如下:
image
image

怀疑是调用代码的这一句有问题,当pixel_values为单个或多个元素的list时,mm['inputs_embeds']都为(1,68,4096)的向量
mm_inputs = prepare_inputs_labels_for_multimodal(llm=llm, input_ids=ids, pixel_values=pixel_values)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the GCG inference call to forward_model and the named prepare_inputs_labels_for_multimodal function. Reproduce the issue using four image inputs and inspect mm['inputs_embeds'], predict, and seg_hidden_states. Done means batched inputs produce outputs with the corresponding batch dimensions rather than single-image shapes.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.