JoyAI-VL-Interaction-Preview produces incoherent outputs in both vLLM and Transformers

## Summary

I can start the official vLLM + vLLM-Omni serving path for `jdopensource/JoyAI-VL-Interaction-Preview`, but the model generates incoherent / random-looking text instead of the expected JoyVL action tokens such as `` or ``.

This does not look like a vLLM-Omni orchestration-only issue, because I can reproduce the same behavior with direct Hugging Face Transformers generation, bypassing vLLM entirely.

Could you please confirm whether the current public Preview checkpoint / tokenizer files are expected to work with the standard `vllm serve` and Transformers loaders, or whether a specific runtime / checkpoint revision / custom loading step is required?

## Model Revision Tested

Hugging Face model:

```text
jdopensource/JoyAI-VL-Interaction-Preview
```

Revision:

```text
ad70e6f0ed63ec1a337bb1c67b82539b42f12d50
```

The local files match the HF metadata / LFS SHA256 values for the four safetensors shards.

## Environment

Hardware:

```text
1x RTX 3090 24GB
```

Runtime:

```text
vllm==0.22.0
vllm-omni==0.22.0
torch==2.11.0+cu130
transformers==5.8.1
```

Repos:

```text
jd-opensource/JoyAI-VL-Interaction commit: 93ff1066a3632c92d1d45e7e81b842baa69cb1d9
vllm-project/vllm-omni commit: fb3abb38fb1916d0a963fe736b22285a7a508b48
```

## Serving Setup

I followed the official vLLM-Omni topology:

1. Plain `vllm serve` for the main model, not `--omni`.
2. `vllm_omni.experimental.fullduplex.joyvl.serving.server` as the interaction orchestrator.

Because I tested on a 24GB RTX 3090, I reduced the memory-related knobs together:

```bash
DTYPE=bfloat16 \
MAX_MODEL_LEN=16384 \
IMAGE_LIMIT=32 \
GPU_MEMORY_UTILIZATION=0.93 \
ENFORCE_EAGER=1 \
bash start_main_vllm.sh
```

The main vLLM service started successfully. Observed memory:

```text
Model loading took about 16.65 GiB
GPU KV cache size: about 21,248 tokens
nvidia-smi after startup: about 20.47GB used, 3.65GB free
```

Then I started the JoyVL orchestrator with memory/delegation disabled for a minimal test:

```bash
CHUNK_FRAMES=32 \
NO_MEMORY=1 \
NO_DELEGATION=1 \
TEMPERATURE=0.0 \
MAIN_BACKEND_URL=http://127.0.0.1:7060/v1 \
bash start_orchestrator.sh
```

The orchestrator also started successfully and returned HTTP 200.

## Actual Output

A minimal image smoke test through the orchestrator returned:

```json
{
"choice": " Timeline还是会垈 :)",
"interaction": {
"action": "response",
"spoke": true,
"text": "Timeline还是会垈 :)",
"delegated_question": null,
"delegation": null,
"chunk_index": 1,
"frame_index": 1,
"inference_skipped": false,
"latency_ms": 7085.0
}
}
```

The same kind of incoherent output happens when calling the main vLLM API directly.

Examples:

```text
Text-only prompt: "Reply with exactly: hello"
Output: "banks java math Gamer math-mar�就可以了 math international爷爷 java ..."
```

```text
JoyVL-style prompt with the official system prompt + one image:
Output: "Timeline还是会垈 :) ... math统统 ..."
```

## Direct Transformers Reproduction

To rule out vLLM / vLLM-Omni as the only cause, I also tested direct Transformers generation with the same checkpoint:

```python
from transformers import AutoProcessor, AutoModelForImageTextToText

processor = AutoProcessor.from_pretrained(model_dir, local_files_only=True)
model = AutoModelForImageTextToText.from_pretrained(
model_dir,
local_files_only=True,
dtype=torch.bfloat16,
device_map="cuda",
attn_implementation="eager",
).eval()
```

Using the JoyVL-style system prompt and one image, direct generation also produced incoherent text:

```text
:"� Exactly Omn我还是uper None_relationship Talesër_relationship Xiao!)
是中国iena Seriously "...都不是没关系…)
...
```

I repeated the direct Transformers check with `torch.float16`; the behavior was essentially the same.

## Logits Check

For the direct Transformers path, I inspected the first-step logits on the JoyVL-style image prompt.

Expected action tokens had extremely low rank:

```text
token id 151669: rank about 109,468, prob about 1.15e-8
token id 151670: rank about 107,240, prob about 1.35e-8
```

The top logits were unrelated normal vocabulary fragments:

```text
:" prob about 0.081
Ribbon prob about 0.041
屠 prob about 0.036
Timeline prob about 0.036
```

I also noticed that the final added-token rows in `model.language_model.embed_tokens.weight` appear to be all zeros locally, while the `lm_head.weight` rows for the action tokens have much smaller norms than ordinary vocabulary rows. I am not sure whether this is expected for this model, but it may be related to the action-token behavior.

## Expected Behavior

For the official JoyVL prompt format, I expected the model to emit one of:

```text

...
... ...
```

or at least produce coherent visual-language text.

## Question

Could you please confirm:

1. Is `ad70e6f0ed63ec1a337bb1c67b82539b42f12d50` the intended working public checkpoint?
2. Are the current tokenizer / added-token files complete for standard vLLM and Transformers loading?
3. Is there a pinned Transformers / vLLM / vLLM-Omni version or special loading step needed to get coherent JoyVL action-token outputs?

Thank you for open-sourcing the project. The serving stack itself starts correctly on a reduced 24GB setup; the blocker here is output quality rather than GPU memory.

Start with start_main_vllm.sh and start_orchestrator.sh, then reproduce the direct Transformers snippet against checkpoint revision ad70e6f0ed63ec1a337bb1c67b82539b42f12d50. Compare the tokenizer, added-token files, model embedding rows, and lm_head rows with the expected action tokens. Done means establishing whether the public checkpoint loads correctly with the documented runtimes or identifying the required revision or loading step.

Written by the indexing model from the issue text.