ByteDance-Seed / ByteDance-Seed/SeedVR
Adapting to custom text encoder
- Dominant language
- Python
- Stars
- 1.4k
- Forks
- 82
- PR merge metrics
- No merged PRs in 30d
Description
Thank you for this awesome work!
I’m currently trying to adapt your model as a baseline for my own training pipeline. Since the text embedding model (as noted in #2) is not open-sourced, I’m in the process of replacing it with a different text encoder.
However, I’m running into difficulties when handling the text embedding format, particularly for batched inputs with varying token lengths. It seems that the nadit forward function expects `txt` and `txt_shape` as inputs. Based on the code [here](https://github.com/ByteDance-Seed/SeedVR/blob/673b56374c2adb21b7628b90df3829d457a52bbb/projects/video_diffusion_sr/infer.py#L293), I constructed a batched text embedding tensor `txt` of shape `[seq_len1 + seq_len2, text_dim]` and the tensor `txt_shape` = `[[seq_len1], [seq_len2]]` for a batch size of 2. However, I encountered the following runtime error:
RuntimeError: The size of tensor a (330) must match the size of tensor b (113) at non-singleton dimension 0.
_(In this case, seq_len1 = 113, seq_len2 = 117)_
From the error and code, I suspect I may be misunderstanding how the model handles text sequences of different lengths within a batch. Could you kindly provide some guidance on the expected formatting or batching strategy for txt and txt_shape? Any clarification would be greatly appreciated.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.