"<image>" is absent in "llava_instruct_150k_zh.jsonl"
Open
- Dominant language
- Python
- Stars
- 5.2k
- Forks
- 448
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 26
Description
Hi,
I noticed that `llava_instruct_150k_zh.jsonl` is used in the [config](https://github.com/InternLM/xtuner/blob/main/xtuner/configs/llava/phi3_mini_4k_instruct_clip_vit_large_p14_336/finetune/llava_phi3_mini_4k_instruct_full_clip_vit_large_p14_336_full_e2_gpu8_internvl_finetune.py) that fine-tunes phi3-based llava using datasets from internvl. However, I found the special token `` is missing from this jsonl file. In the current llava pipeline, image embeddings won't be inserted into the input sequence of LLM if this special token is absent.
Contributor guide
Assessment
This issue has not been assessed yet.