ByteDance-Seed / ByteDance-Seed/Bagel
多batch推理
- Dominant language
- Python
- Stars
- 6.2k
- Forks
- 545
- PR merge metrics
- No merged PRs in 30d
Description
bagel在源代码当中其实提供了多batch的处理流程,但是不知道出于何种原因,这个处理流程在发布的时候被藏起来了。模型处理多batch的思路很成熟,把文本token和图像token交叉放置到同一个序列里面,然后一起放到LLM当中推理,推理结束之后再从序列里面把数据取出来。
原版设计思路可以参见`eval/gen/gen_images_mp.py`,具体的应用方式如下(以文生图为例):
1、准备多段文本,和**每张图片的尺寸**。bagel处理的时候会按照图片尺寸参数的数量来决定需要同时生成几张,多余的部分会被直接丢弃。
2、对`inferencer.py`当中的`init_gen_context`函数做如图的修改:
。
然后直接把数据塞进interleave_inference函数就可以了。
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with eval/gen/gen_images_mp.py to understand the existing multi-batch flow, then inspect init_gen_context in inferencer.py and its use by interleave_inference. Confirm how multiple text prompts and image sizes determine the batch count, including disposal of extra inputs. Done means the published interface can perform multi-batch inference as described.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100