The authors propose a diffusion language model that represents text as a continuous diffusion process over fixed-width binary bitstreams, aiming to close the gap with autoregressive models. Each semantic token is represented as an analog bit sequence, and the model uses a matched-filter residual parameterization to separate contextual learning from the independent-bit posteriors. Crucially, they introduce a stochastic sampler that applies Langevin-type corrections gated by the entropy-rate profile—concentrating randomness in high-information regions while remaining nearly deterministic elsewhere. On LM1B, their 130M-parameter model reaches a generative perplexity of 59.76 (matching the autoregressive reference) using 256 function evaluations; on OpenWebText, it achieves 27.06 perplexity with 4× fewer steps than prior 1024-step baselines. Bitstream diffusion also removes the vocabulary-size scaling bottleneck, predicting O(log V) bitwise logits instead of O(V) token logits.
Main takeaways:
- Represents text as a diffusion process over continuous binary bitstreams (fixed-width analog bits) rather than discrete tokens.
- Stochastic sampler gates Langevin corrections by entropy rate, concentrating randomness where information is high and being deterministic elsewhere.
- On LM1B: 130M-param model hits generative perplexity 59.76 (matching autoregressive baseline) in 256 neural function evaluations.
- On OpenWebText: achieves perplexity 27.06 with 4× fewer steps than prior 1024-step diffusion baselines.
- Removes O(V) vocabulary scaling bottleneck by predicting O(log V) bitwise logits, reducing memory and increasing throughput.