ByteDance-Seed / ByteDance-Seed/SeedVR

Adapting to custom text encoder

Open
#14 3 comments 1 reaction 0 assignees View on GitHub
Dominant language
Python
Stars
1.4k
Forks
82
PR merge metrics
No merged PRs in 30d

Description

Thank you for this awesome work!

I’m currently trying to adapt your model as a baseline for my own training pipeline. Since the text embedding model (as noted in #2) is not open-sourced, I’m in the process of replacing it with a different text encoder.

However, I’m running into difficulties when handling the text embedding format, particularly for batched inputs with varying token lengths. It seems that the nadit forward function expects `txt` and `txt_shape` as inputs. Based on the code [here](https://github.com/ByteDance-Seed/SeedVR/blob/673b56374c2adb21b7628b90df3829d457a52bbb/projects/video_diffusion_sr/infer.py#L293), I constructed a batched text embedding tensor `txt` of shape `[seq_len1 + seq_len2, text_dim]` and the tensor `txt_shape` = `[[seq_len1], [seq_len2]]` for a batch size of 2. However, I encountered the following runtime error:

RuntimeError: The size of tensor a (330) must match the size of tensor b (113) at non-singleton dimension 0.
_(In this case, seq_len1 = 113, seq_len2 = 117)_

From the error and code, I suspect I may be misunderstanding how the model handles text sequences of different lengths within a batch. Could you kindly provide some guidance on the expected formatting or batching strategy for txt and txt_shape? Any clarification would be greatly appreciated.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.