[Question] Disaggregated training (producer:consumer 1:7) ~3.2x slower than DP for Qwen3.6-35B-A3B — expected?

Checklist

• 1. If the issue you raised is not a feature but a question, please raise a discussion at https://github.com/sgl-project/SpecForge/discussions/new/choose Otherwise, it will be closed.
• 2. Please use English, otherwise it will be closed.

Motivation

I'm training a draft model for Qwen3.6-35B-A3B using SpecForge and observing a significant wall-clock time difference between the disaggregated topology and the original DP (data-parallel) approach. I'd like to understand whether this gap is expected, and if there are recommended tuning strategies.

Setup

Observation

The disaggregated approach is roughly 3.2× slower in wall-clock time compared to the original DP method for the same dataset and epoch count.

Questions

1. Is this magnitude of slowdown expected for the disaggregated topology on a single node with a 1:7 producer-to-consumer ratio?
2. What are the primary bottlenecks — is it the feature capture throughput on the producer side, the Mooncake transfer overhead, or consumer-side waiting/starvation?
3. Are there recommended producer:consumer ratios or runtime tuning knobs (e.g., producer_lease, in_flight_high_watermark, in_flight_low_watermark) that can significantly close this gap?
4. Would using multiple capture servers (repeating server_urls) or offline pre-captured features be the recommended path for this model size?
5. Any known scaling behavior or best-practice configs for Qwen3.6-35B-A3B disaggregated training?

Additional context

• I'm still experimenting with different producer:consumer ratios; 1:7 is the current setting.
• The DP baseline uses the same hardware and dataset, just with the colocated target-inference + trainer approach.
• Happy to provide logs, throughput stats, or config YAML if helpful.

No source files or tests are named. Start by reviewing the disaggregated training configuration and runtime settings mentioned in the issue, then compare producer throughput, Mooncake transfer overhead, and consumer waiting using logs, throughput stats, and config YAML; done means establishing whether the slowdown is expected and documenting validated ratios or tuning guidance.