• TRM (USD/COP)COP 3,461.92 +1.37% +COP 46.82Colombia's official exchange rate (Financial Superintendence), valid Nov 21 to Nov 23
  • EuroCOP 3,804.64 +1.37% +COP 51.45Reference: TRM crossed with the European Central Bank rate of Nov 20; not the market quote
  • Mexican pesoCOP 181.56 +1.37% +COP 2.45Reference: TRM crossed with the European Central Bank rate of Nov 20; not the market quote
← Back to the community
Technology

Why are diffusion LLMs like Inception's Mercury replacing sequential token decoding to top 1,000 tokens per second?

Discrete diffusion large language models led by Inception Labs' Mercury architecture and open-weight LLaDA are shattering the sequential memory-bandwidth wall of autoregressive Transformers, refining entire blocks of tokens in parallel across iterative denoising passes to exceed one thousand tokens per second on standard enterprise GPUs.

When we profile traditional autoregressive models like GPT and Claude during inference, text generation is fundamentally memory-bound: because token N cannot be computed until token N-1 is finalized, GPU compute units sit idle waiting to read massive model weights from High Bandwidth Memory for every single word. Discrete diffusion LLMs invert this paradigm by adapting the iterative denoising math of image generators like Stable Diffusion to discrete text tokens. Starting with a canvas of masked noise tokens, a diffusion LLM predicts and refines dozens or hundreds of tokens simultaneously in a few parallel forward passes.

By turning sequential decoding into parallel matrix operations that saturate GPU Tensor Cores, Inception Labs' Mercury 3 achieves over 1,000 tokens per second on Nvidia H100 hardware across a one-million-token context window—running five to ten times faster than speed-optimized autoregressive counterparts. Beyond raw throughput for real-time voice agents and IDE code completion, bidirectional diffusion refinement allows the model to globally plan and enforce strict JSON schemas or syntactic constraints across an entire output block without left-to-right hallucinations.

Will parallel diffusion LLMs overtake sequential autoregressive decoding for latency-critical AI agents, or will speculative decoding keep traditional Transformers ahead?

Where will discrete diffusion LLMs (dLLMs) have the biggest impact?

0 votes

Comments

Be respectful. Comments reported by several people are hidden until a moderator reviews them.

You do not need an account to read, only to comment. Sign in with a code sent to your email or with your Google account. We only show the name you choose, never your email.