ByteDance-Seed / ByteDance-Seed/Bagel

Support for Text-Image Interleaved Output?

Open
#99 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
6.2k
Forks
545
PR merge metrics
No merged PRs in 30d

Description

Thanks for your brilliant work on Bagel—it's an impressive and inspiring project!

I’m wondering whether Bagel currently supports (or plans to support) text-image interleaved output, as described in Section 3.3 "Vision-Text Interleaved Data" of the paper.

For example, can the model perform inference like the following:

Input:
```
Use pictures and words to explain each step to cook eggs.
```

Output:
```
[image1]
Step 1: Crack the eggs into a bowl and whisk them together with a fork or a whisk.
[image2]
Step 2: Heat a non-stick pan or skillet over medium heat.
[image3]
Step 3: Once the oil is hot, pour the eggs into the pan and use a spatula to gently scramble them.
[image4]
Step 4: Once the eggs are cooked to your desired doneness, season with salt and pepper to taste.
[image5]
Step 5: Once the eggs are cooked to your liking, transfer them to a serving plate and enjoy!
```

If this isn't currently supported, I’d be curious to know whether it's feasible to extend Bagel to this use case, and if you have any recommendations on how to approach it with the existing codebase.

Thanks again for your amazing work—really excited to see how this project evolves!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading Section 3.3, “Vision-Text Interleaved Data,” of the referenced paper and compare its requirements with Bagel’s current multimodal inference behavior. The issue names no files or tests; completion would require a scoped design and a demonstrated decision about whether interleaved text-image output is supported or feasible.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.