NVIDIA-NeMo / NVIDIA-NeMo/Export-Deploy
Ray Deployment of moonlight 16B (MBridge) fails
Open
@oyilmaz-nvidia is already working on this.
Since Mar 2, 2026.
bug
- Dominant language
- Python
- Stars
- 42
- Forks
- 18
- Avg merge
- 1d 2h
- Merged PRs (30d)
- 8
Description
Describe the bug
(ServeReplica:megatron_model:MegatronRayDeployable pid=836, ip=100.65.137.96) File "/opt/Megatron-Bridge/3rdparty/Megatron-LM/megatron/core/transformer/transformer_layer.py", line 609, in _forward_attention [repeated 3x across cluster]
(ServeReplica:megatron_model:MegatronRayDeployable pid=836, ip=100.65.137.96) attention_output_with_bias = self.self_attention( [repeated 3x across cluster]
(ServeReplica:megatron_model:MegatronRayDeployable pid=836, ip=100.65.137.96) ^^^^^^^^^^^^^^^^^^^^ [repeated 3x across cluster]
(ServeReplica:megatron_model:MegatronRayDeployable pid=836, ip=100.65.137.96) self.config.cache_mla_latents [repeated 3x across cluster]
(ServeReplica:megatron_model:MegatronRayDeployable pid=836, ip=100.65.137.96) AssertionError: currently to use dynamic backend for MLA cache mla latents must be true [repeated 3x across cluster]
Steps/Code to reproduce bug
- ToT MBridge/MCore
- Moonlight 16B pretrain checkpoint
- TP1/PP1/CP1
Expected behavior
Can deploy a ray cluster
Additional context
Add any other context about the problem here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.