explore diffusion llm

Diffusion language models (DLMs) have recently attracted significant attention across both academic and industry communities, with notable deployments including Google’s Gemini Diffusion and Inception Labs’ Mercury and Meta's recent contribution, EditFlows. These models represent a fundamentally different approach to text generation, moving away from the traditional autoregressive models.

Diffusion language models work by corrupting sequences of coherent text to noisy versions of it and training a model to approximate the reverse corruption process (denoising). In contrast to autoregressive models, which generate tokens sequentially, diffusion language models leverage parallel decoding strategies, thereby presenting a promising approach to significantly reducing inference time (InceptionLabs reports a 10x speedup in decoding with Mercury). The number of generation steps is defined by a fixed denoising schedule, which can be considerably smaller than the length of the output sequence.

This paradigm presents several advantages: it enables bidirectional context modeling, improves controllability over generation (e.g., for editing or conditioning tasks), and demonstrates competitive performance with autoregressive counterparts across a range of benchmarks.

Earlier work, including the D3PM framework and other studies, established the mathematical formulation for discrete diffusion models, defining both the corruption and denoising dynamics. Recent research has demonstrated the scalability of diffusion language models up to 1B parameters. To date, the largest masked diffusion models are 7B Dream and 8B LLaDa. Recently, a survey on diffusion models for language models was released.

In this work, we will explore whether diffusion language models are scalable and promising for the project:

• literature review
• run experiments with existing models (Dream and LLaDA)
• exploring scaling laws for diffusion language models.