The authors propose BitLM, a language model that generates multiple tokens in parallel by representing each token as a binary code and using diffusion to denoise several tokens at once within each block. Unlike pure diffusion models, BitLM preserves left-to-right causal attention across blocks while making joint decisions within blocks, combining the reliability of autoregressive modeling with the speed of parallel generation. This approach replaces the expensive vocabulary softmax with bitwise denoising, enabling faster pre-training and inference.
Main takeaways:
- Standard LLMs generate one token at a time, but natural language often carries meaning in multi-token units (phrases, collocations)
- BitLM represents tokens as fixed-length binary codes and denoises multiple tokens in parallel within each block
- Maintains causal left-to-right attention across blocks, so it's still a proper language model, not just a diffusion model
- Replaces the large-vocabulary softmax (expensive) with bitwise denoising in a compact binary space
- Achieves faster inference and stronger performance by treating token generation as iterative commitment in binary space rather than one-hot selection