Discrete Diffusion Reading Group

In today's session, Mohsin Hasan and Viktor Ohanesian present their recent work on Discrete Feynman-Kac Correctors, a framework for controlling the sampling distribution of discrete diffusion models at inference time. The method uses Sequential Monte Carlo (SMC) to enable temperature control (annealing), combine multiple diffusion processes, and incorporate external reward functions, without retraining or fine tuning the original model. The framework is demonstrated on applications including Ising model sampling, improved code generation, and reward guided protein sequence generation.

In today's session, Andre He (LTI @ CMU) presented his paper Reasoning with Latent Tokens in Diffusion Language Models, exploring why diffusion language models outperform autoregressive (AR) models on synthetic reasoning tasks. Andre argues that diffusion models naturally maintain "latent tokens" (as joint predictions over undecoded positions) which enable planning and lookahead, and demonstrated that modulating these latent tokens creates a smooth tradeoff between inference speed and output quality. Andre further showed that introducing a similar multi-token prediction objective into AR models significantly improves their reasoning performance, suggesting latent tokens as a general mechanism for enhancing global coherence.

Dimitri von Rütte (ETH) and Zhihan Yang (Cornell) present "Scaling Behavior of Discrete Diffusion Language Models" (https://arxiv.org/abs/2512.10858) and "Scaling Beyond Masked Diffusion Language Models" (https://www.arxiv.org/abs/2602.15014), two recent papers presenting systematic scaling laws of uniform-state and hybrid discrete diffusion LLMs. Importantly, both papers challenge the dominance of Masked Diffusion.