ByteDance-Seed / ByteDance-Seed/Bagel

多batch推理

Open
#291 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

bagel在源代码当中其实提供了多batch的处理流程,但是不知道出于何种原因,这个处理流程在发布的时候被藏起来了。模型处理多batch的思路很成熟,把文本token和图像token交叉放置到同一个序列里面,然后一起放到LLM当中推理,推理结束之后再从序列里面把数据取出来。

原版设计思路可以参见`eval/gen/gen_images_mp.py`,具体的应用方式如下(以文生图为例):

1、准备多段文本,和**每张图片的尺寸**。bagel处理的时候会按照图片尺寸参数的数量来决定需要同时生成几张,多余的部分会被直接丢弃。
2、对`inferencer.py`当中的`init_gen_context`函数做如图的修改:
Image

然后直接把数据塞进interleave_inference函数就可以了。

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with eval/gen/gen_images_mp.py to understand the existing multi-batch flow, then inspect init_gen_context in inferencer.py and its use by interleave_inference. Confirm how multiple text prompts and image sizes determine the batch count, including disposal of extra inputs. Done means the published interface can perform multi-batch inference as described.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.