MDP & Timestep RL for Discrete Diffusion LLMs

Wed, 30 Sep 2026 15:44:00 +0900

Semi-AR (Block AR); Agentic multi turn; Only terminal/outcome reward, \(\gamma=1\) as the convention for LLM RL.

Setup

Index

Some canvas with idx \(c\) (length \(C\)) includes denoising timestep index (not token index) \([n_c, n_c+T_c)\), \(T_c\) is not constant over \(c\) by early-stopping. Global denoising step \(n\).

From \(n\), a tuple of indices can be specified

\[ n\mapsto (j(n),c(n),t(n)) \]

\(j(n)\): (global) turn index
\(c(n)\): global canvas index
\(t(n)\): timestep index in the canvas

thus

\[ n=\sum_{c< c(n)}T_c+t(n). \]

State

For some timestep \(n\), define its realized state \(s_n\) (and r.v. \(S\))

Formulation with dLLMs on itu

MDP & Timestep RL for Discrete Diffusion LLMs

Setup

Index

State