Inference speed (~14–15 TPS) with Molmo-7B-D-0924 (4-bit) and no improvement using vLLM

Hi team,

I’m currently running inference with Molmo-7B-D-0924 and observing relatively low throughput (~14–15 tokens per second).

Setup

Model: Molmo-7B-D-0924 (fine-tuned checkpoint)

Quantization: 4-bit (bitsandbytes)

Inference type: Image + text prompt (multimodal generation)

Decoding: Standard generate() API

Measured throughput: ~14–15 TPS

vLLM Attempt

I also tried installing and running inference through vLLM (pip install vllm) to see if I could improve decoding throughput.

However:

There are transformer architecture mismatches between vLLM and Molmo’s multimodal wrapper.

After working around loading issues, I still observe roughly the same throughput (~14 TPS).

This suggests that the bottleneck may not be standard transformer decoding, but potentially the multimodal pipeline (vision encoder + fusion).

Questions

Is ~14–15 TPS expected for 4-bit multimodal inference with Molmo-7B-D-0924?

Are there recommended optimizations for increasing inference throughput?

FlashAttention support?

torch.compile compatibility?

Precomputing / caching vision embeddings?

Any internal optimizations used for the demo model?

Is there an officially supported or recommended way to run Molmo with vLLM?

Are there plans for an optimized inference path (e.g., vLLM-compatible export or TensorRT support)?

I’m aiming to increase throughput toward ~50 TPS if possible, but I’m unsure whether that is realistic given the current architecture.

Any guidance would be greatly appreciated.

Thanks!