What does a discrete diffusion model learn?

This post accompanies our paper “What Does a Discrete Diffusion Model Learn?”. It is a guided summary of the paper’s main results.

The question

Discrete diffusion models generate language and other categorical data by corrupting sequences directly on their alphabet and learning to run the corruption backward. If you read the literature, the object being learned goes by several names: a denoiser that predicts the clean token, a concrete score that predicts ratios of noisy marginals, a bridge plug-in that averages the forward kernel under the network’s prediction. These are widely treated as interchangeable descriptions of the same model, but they are not.

A neural network’s output is not a generative process until you specify how it is converted into reverse jump rates, and reading the same output in the wrong coordinate changes the process you are training and sampling from. This, far from a purely academic observation, has real practical implications:

• In uniform (UDLM) and GIDD diffusion, the standard bridge plug-in objective does not train a denoiser. It trains a cavity law: a predictor that uses the noisy context but ignores the noisy token it is about to revise.
• A raw denoiser head under UDLM noise makes the training objective diverge at initialization; the folklore that denoisers “misbehave” outside masked diffusion has a precise cause.
• Masked diffusion hides all of this, because at a masked position the local observation carries no information, and the denoiser and cavity coincide, but that coincidence is special to masked diffusion rather than a general property of other diffusion processes.

This exposes a fundamental question underlying our current understanding: what does the training objective of a discrete diffusion model actually optimize? The paper’s answer has two levels: what the objective is (an exact distance between path laws), and what the learned object is (one reverse rate with three coordinate systems).

Continuous-time discrete diffusion

Diffusion models turn sampling from a data law $\pdata$ into the problem of reversing a noising process. Choose a forward process $(Z_t)_{t\in[0,T]}$ whose marginals $q_t$ interpolate between $q_0 = \pdata$ and an easy-to-sample terminal law $q_T$ and then learn to run it backward. For discrete data the natural forward process is a continuous-time Markov chain (CTMC): a jump process on a finite state space, characterized by its infinitesimal generator $Q_t(x,y)$, the instantaneous rate of jumping from $x$ to $y$ at time $t$. It can be computed as the time derivative of the corruption kernel $q_{t|0}(y|x)$.

Two reverse-time rates matter in discrete diffusion. The clean-conditioned reverse rate (the “bridge”), which knows the clean data $z_0$, and the marginal reverse rate, which does not:

$$\RevQ_t(z, y \given z_0) = Q_t(y, z)\,\frac{q_{t|0}(y \given z_0)}{q_{t|0}(z \given z_0)}, \qquad \RevQ_t(z, y) = Q_t(y, z)\,\frac{q_t(y)}{q_t(z)}.$$

The first is easy to compute: we know the corruption kernel $q_{t|0}$ so the above is a simple substitution. The second is what generation actually needs, since only the reverse CTMC with the marginal rates transports $q_T$ back to $\pdata$. The model’s job is to induce reverse rates $\RevQ^\theta_t(z_t, y)$, functions of the noisy state only, but not of the clean data $z_0$ (which will of course not be available during sampling).

The negative ELBO objective

The training objective is the diffusion negative ELBO, inherited from the variational autoencoder bound with the entire noising trajectory as the latent variable. (In the paper we derive the continuous-time ELBO twice: as an infinitesimal limit of the discrete-time bound, formalizing the typical informal argument often used in the literature, and via a finite-state Girsanov argument.) The continuous-time ELBO has an exact per-time form given by the local rate divergence

$$\Phi(a, b) = a \log\frac{a}{b} - a + b,$$

a Bregman divergence that measures the KL rate between two jump processes. From this, the full-interval ELBO is the integral:

$$\mathrm{ELBO}_{[0,T]}(\theta; z_0) = -\int_{0}^{T} \E_{q_{t|0}(z_t \given z_0)} \Big[ \textstyle\sum_{y \neq z_t} \Phi\big(\RevQ_t(z_t, y \given z_0),\; \RevQ^\theta_t(z_t, y)\big) \Big]\, dt.$$

However, it is uncommon to train on the full $[0,T]$ interval. Endpoint behavior at $t = 0, T$ can be singular or numerically unstable, posterior ratios have high variance there, and Monte Carlo sampling uses different time windows and importance-sampling schemes. In practice one instead clips to a window $[t_1, t_2] \subset (0, T)$. In the paper, we prove that:

Here $p^\theta_{0|t_1}$ is the final reconstruction decoder (the final hop from the mildly noisy state $z_{t_1}$ back to clean data, often done using the same network head) and $p^\theta_{t_2}$ is the terminal prior from which sampling starts. Versions of the path term are standard across the literature, but in this paper we insist on keeping all the terms, both the two boundary terms as well as any terms of $\Phi$ that do not depend on the model, which most presentations instead drop as they do not matter for optimization. However, doing so hides what the ELBO objective actually is, and even worse, once windows are clipped and boundary terms dropped, it breaks the comparison of reported values across noising processes. As we show in the paper, the boundary terms are exactly what restores comparability, what enables the path-law interpretation of the ELBO, and what validates the calibration identities below.

Still, there is a well-known caveat about the ELBO, including its continuous-time version above: it is only a bound on the model’s likelihood, and the gap to the object of interest is unknown. If two papers report different values, is the better number a better generative model or merely a tighter bound? And there is another source of potential confusion: the loss regresses against the clean-conditioned rate, but generation needs the marginal rate. So what is this ELBO optimizing for?

The Oracle Distance: the ELBO is the distance to the oracle

The answer compares whole trajectories, whose formalization is a path law: a starting endpoint law plus jump rates, which uniquely determine the distribution of the entire trajectory. Two path laws matter for us: $P^\star$, the law of the noising trajectories themselves, defined by reading backward from the terminal $q_{t_2}$ using the marginal reverse rates (the oracle); and $P^\theta$, the model’s law, run backward from our terminal of choice $p_{t_2}^\theta$ with the learned reverse rates $\RevQ^\theta$. In the clean full-interval setting the paper then proves an exact identity that we call the Oracle Distance:

The negative ELBO is, after separating a constant equal to the data entropy, exactly the path-law KL divergence between the true or oracle reverse trajectory and the learned one. Thus, training a discrete diffusion model is path matching to the oracle!

This result also makes precise what the gap between the ELBO and the model likelihood is. Since marginalizing a path law down to its endpoint cannot increase relative entropy, by the usual information-bottleneck inequality, we get

$$\KL\big(\pdata \,\|\, p^\theta_0\big) \;\le\; \KL\big(P^\star \,\|\, P^\theta\big) = -H(q_0) - \E_{q_0}\big[\mathrm{ELBO}_{[0,T]}(\theta; z_0)\big].$$

This explains both the power and the limitation of training on the negative ELBO. On the one hand, optimizing the ELBO makes the model’s entire reverse trajectory follow the oracle’s, and as a consequence makes the distribution of generated samples close to the target distribution. On the other hand, the path KL can be substantially larger than the endpoint KL, that is, the ELBO may penalize pathwise discrepancies that are irrelevant for the final samples. That redundant penalization is precisely the gap between the ELBO and the model likelihood.

The oracle identity is another point where keeping the boundary terms is crucial for the identity to survive window clipping. In the paper we then show

Potentially confusing on first read is the fact that the reconstruction term at $t_1$ sits outside the path KL, while the usual terminal-prior mismatch at $t_2$ sits inside it. This comes out of the finite-state Girsanov formula: a KL between path laws decomposes as the KL between their laws at the starting time plus an integral of local rate divergences,

$$\KL\big(P^\star_{[t_1,t_2]} \,\big\|\, P^\theta_{[t_1,t_2]}\big) = \KL\big(q_{t_2} \,\big\|\;p^\theta_{t_2}\big) + \int_{t_1}^{t_2}\E_{q_t}\Big[\textstyle\sum_{y\neq z_t}\Phi\big(\RevQ_t(z_t,y),\;\RevQ^\theta_t(z_t,y)\big)\Big]\,dt.$$

Two consequences fall out immediately from the Oracle Distance identity:

1. The optimizer is a projection. The path KL vanishes exactly at the marginal reverse rates, so among all rates that depend only on the noisy state, the unique optimum is

The training target is clean-conditioned, but its conditional expectation given what the model can see (i.e., the average over $z_0$ given the noisy state $z_t$) is the marginal rate needed for generation. This is the discrete-diffusion counterpart of the conditional-mean principle behind flow matching in continuous-diffusion. It is also directly visible in a trained network:

2. The ELBO has a universal floor. Since the KL term is non-negative and achievable (for an expressive enough model class),

The best achievable value does not depend on the noising process or the parameterization. It is the entropy of the data, and the excess above it is precisely the path KL mismatch to the true reverse process. This makes negative ELBO values interpretable and comparable: after subtracting the entropy, they all measure a genuine divergence, not just a bound of unknown tightness.

Every noising process carries the same oracle cost, but on a different information schedule

Where does the entropy floor come from? Intuitively, it must come from the error that the oracle itself has to make given it lacks access to the actual clean data $z_0$. It turns out that the oracle’s per-time ELBO is exactly the rate at which the forward process destroys information:

$$\mathcal{J}^\star_t := \E\Big[\textstyle\sum_{y} \Phi\big(\RevQ_t(Z_t, y \given Z_0), \RevQ_t(Z_t, y)\big)\Big] = \frac{d}{dt} H(Z_0\given Z_t) \;\ge\; 0,$$

a CTMC counterpart of the I-MMSE relation. Integrated over time, this telescopes to the total information the data carries: $H(q_0)$. Every noising process destroys the same total; they differ only in when they do so.

The figure below shows this, evaluating the oracle cost $\mathcal{J}^\star_t$ and its integral using an exactly-solvable toy model (a Markov chain over $V = 8$ tokens, length $L = 8$, with $H(q_0)/L \approx 1.25$ nats). Dragging $\lambda$ interpolates masked ($\lambda = 0$) and uniform ($\lambda = 1$) through the GIDD family (via a simple mixing of the terminal laws between a masked and a uniform distribution). While the oracle cost $\mathcal{J}^\star_t$ over time may change, its total area $\int_{0}^{T}\mathcal{J}^\star_t\,dt$, that is, the negative ELBO, is always the same: the data entropy.

GIDD λ

0 · masked1 · uniform

λ = 0.50

(a) information destroyed, H(Z₀ | Zₜ) (per-token)

(b) destruction rate, 𝒥ₜ^⋆ = d/dt H(Z₀ | Zₜ) (per-token)

The second mechanism behind the Oracle Distance identity is a Pythagorean split: because $\Phi$ is a Bregman divergence, the negative ELBO cost at each time separates with no cross term into what the oracle pays plus how far the model is from the oracle,

$$\underbrace{\E\Big[\textstyle\sum_{y} \Phi\big(\RevQ_t(Z_t, y \given Z_0), \RevQ^\theta_t(Z_t, y)\big)\Big]}_{\text{negative ELBO per-time cost}} = \underbrace{\;\mathcal{J}^\star_t\;}_{\text{oracle cost}} + \underbrace{\E\Big[\textstyle\sum_{y} \Phi\big(\RevQ_t(Z_t, y), \RevQ^\theta_t(Z_t, y)\big)\Big]}_{\text{model–oracle path divergence}}.$$

From this and the information-loss-rate identity for the oracle, the main step to obtain the Oracle Distance expression for the ELBO (in the case with no boundary terms) is a simple integration over time: the oracle cost telescopes to $H(q_0)$ while the divergence accumulates into the path KL.

A consequence of this split is that, at a diffusion time $t$, there is a fixed best-achievable floor of the local ELBO, and training can only close the gap to that floor but never go below it. This is a powerful tool to diagnose whether, and if so where, training is improvable or already optimal, and it leads to heuristics to guide where to place more modeling effort, or how to distribute the Monte Carlo samples and thus compute when estimating the ELBO during training.

All results so far are completely general: they hold for any noising process and of course any parameterization of the model reverse rates:

Before moving on to sequence modeling, a final neat observation is that using a particular importance-sampling (IS) schedule for the Monte Carlo estimation of the ELBO, as is commonly done, is equivalent to changing the clock on which the noising process runs: the IS weighting, read as a Jacobian change-of-measure, goes inside $\Phi$ in the ELBO integrand and gives the reverse rates of the process in the reparameterized time. This observation connects nicely to the oracle rate result, $\mathcal{J}^\star_t = -\tfrac{d}{dt}I(Z_0 ; Z_t)$: an IS schedule proportional to $\mathcal{J}^\star_t$ is equivalent to choosing as the process’ time $\tau$ the residual mutual information between clean and noisy data, so that $I(Z_0; Z_\tau) = (1 - \tau)\,H(q_0)$. On this information-uniform clock, every noising process straightens onto the same diagonal in the left panel of the interactive figure above, while all $\mathcal{J}^\star_t$ in the right panel become the same constant.

There might be an argument to be made for using such a clock also for reverse sampling, at least when the model is close to the oracle: it would mean “adding information back” at a constant rate. However, the optimal IS schedule in terms of minimizing the Monte Carlo variance of the ELBO is in general not the one linear in information loss, but one proportional to $\sqrt{\E\Big[\mathcal{J}^\theta_t(z_0,z_t)^2\Big]}$.

Sequence modeling: three coordinates, one object

Everything so far holds for any finite state space. Sequence modeling only adds structure: the state space is a product $\mathcal{X}_1 \times \cdots \times \mathcal{X}_L$, and the standard noising processes corrupt each position independently, so jumps occur one token at a time. Computing the oracle projection for such processes, the true reverse rate at position $i=1,\ldots,L$ turns out to depend on the data only through the clean-token posterior marginals $q(z_0^i \given z_t)$, never through full-sequence laws. This single fact is what makes token-level parameterizations sufficient and discrete diffusion tractable.

But now a practitioner meets three apparently different options to map an NN head to reverse rates, and here is the central result: they are three exact coordinate systems for the same object (the true reverse rate), each of them uniquely optimized at a particular oracle law and with closed-form conversions between them. Denoting by $z_t^{-i}$ the sequence $z_t$ excluding the $i$-th token, the situation is summarized by the following diagram:

The three laws:

• The denoiser $\pi_i^\star(z_0^i \given z_t) = q(z_0^i \given z_t)$: the clean-token posterior given the full noisy sequence. Fitting a head to it and averaging the forward-kernel ratio under it gives the reverse rate. This is the classic mean parameterization (D3PM, MDLM, MD4).
• The cavity $\mu_i^\star(z_0^i \given z_t^{-i}) = q(z_0^i \given z_t^{-i})$: the clean-token posterior given the noisy context only, ignoring the local noisy token. The bridge plug-in used by UDLM, GIDD, and Duo is optimized by this law, not by the denoiser. The name comes from the leave-one-out “cavity” constructions of statistical physics.
• The score $s_i^\star(z_t^i, y^i \given z_t^{-i}) = q_t(y^i, z_t^{-i})/q_t(z_t)$: the clean-data-free ratio of noisy marginals (SEDD, RADD, TCSM), which enters the rate directly.

All three agree at the optimum: each head is optimized exactly at its own oracle law, and the coordinates are not interchangeable, so that we cannot feed the law of one coordinate into the sampling formula of another. This has nothing to do with learning: it also happens for the exact oracle laws:

Why masked diffusion is special

The denoiser and the cavity are linked by a simple Bayes update:

$$\pi_i^\star(z_0^i \given z_t) \;\propto\; \mu_i^\star(z_0^i \given z_t^{-i}) \; q_{t|0}^i(z_t^i \given z_0^i).$$

That is, the token-wise forward kernel $q_{t|0}^i(z_t^i \given z_0^i)$ is precisely the factor that turns a posterior average given the noisy context $z_t^{-i}$ into one that also observes the local noisy token $z_t^i$. So the information about the clean token splits in two: what the context reveals (the cavity, which must be learned) and what the local token adds (the forward kernel, available analytically). In general, a denoiser does not make use of this analytically available part and instead has to learn both at once, a concrete source of training difficulty. Further, near the clean endpoint the kernel ratios blow up, and this makes things even worse: the denoiser’s UDLM ELBO diverges at initialization.

In masked diffusion, a corrupted token is always the mask symbol, so the local observation is uninformative: the Bayes factor is constant, and $\pi_i^\star = \mu_i^\star$. Thus, the denoiser–cavity distinction vanishes identically, which is why masked diffusion tolerates conflating the two.

The framework also yields a debugging tool. An “uninformative” cavity head such that $\mu^\theta \equiv 1/V$ has per-token negative ELBO exactly $\log V$: for any noising process and any training window, provided boundary terms are kept. If your implementation does not reproduce $\log V$ at initialization, something is wrong!

What the objective cannot see

The framework cleanly separates the two sources of generative error of a trained model. The first is reverse-rate error: the gap between the model’s rates and the true ones. By the Oracle Distance, this is exactly what the ELBO measures. The second is factorization error: parallel ancestral samplers update sequence positions independently within a step, while the true finite-step reverse kernel does not factor in this way. This error is invisible to the ELBO, and it exists even for a model with exactly zero reverse-rate error:

Practical takeaways

The theory compresses into four rules:

1. Match the head to its loss at sampling time or apply the right coordinate conversion. Otherwise you are using the wrong sampling formula.
2. Avoid a denoiser head for uniform and GIDD diffusion. Its ELBO diverges at initialization, while a cavity head stays finite.
3. Use the $\log V$ identity as a unit test. An uninformative cavity head must give per-token negative ELBO $\log V$, independent of process and window.
4. Keep the boundary terms. Reconstruction and terminal-prior terms are required for fair ELBO comparisons across processes, the ELBO calibration results, and the Oracle Distance interpretation.

Conclusion

The answer to the title is then: a discrete diffusion model learns the marginal reverse jump rate, given by the posterior average of the clean-conditioned reverse rate, and its training loss is the data entropy plus the path-law KL to this oracle reverse process. Denoisers, bridge plug-in predictors, and concrete scores are coordinates of that one object when the underlying state is a product-space and the noising process factors across positions.

Everything here is verified on an exactly-solvable model where every oracle quantity is available in closed form; the natural next step is confirming that the coordinate-conversion gains persist at scale. If you work on discrete diffusion, the paper’s Section 2 is a self-contained version of this post with the precise statements, with Sections 4–8 carrying out all the proofs in detail.

@misc{casadonoguerales2026discretediffusion,
  title         = {What Does a Discrete Diffusion Model Learn?},
  author        = {Casado Noguerales, Rodrigo and Sch{\"o}lkopf, Bernhard
                   and Hofmann, Thomas and Raoufi, Aran},
  year          = {2026},
  eprint        = {2607.05381},
  archivePrefix = {arXiv},
  primaryClass  = {cs.LG},
  url           = {https://arxiv.org/abs/2607.05381},
}