The authors argue that evaluating text with standard autoregressive language models introduces positional bias—early tokens are scored with less context than later ones—and propose DiffScore, which uses masked diffusion models to score every token with full bidirectional context. By measuring how well text can be reconstructed at different masking rates, DiffScore creates a quality hierarchy from local fluency to global coherence and provides diagnostic tools like multi-timestep profiles and bidirectional PMI decomposition that separate fluency from faithfulness.
Main takeaways:
- Autoregressive evaluation gives early tokens less context, conflating architectural asymmetry with actual text quality
- DiffScore uses masked reconstruction at varying masking rates, so every token is scored with full bidirectional context
- Multi-timestep profiles show how quality changes across masking rates, revealing whether problems are local (fluency) or global (coherence)
- Bidirectional PMI decomposition separates fluency (word choice) from faithfulness (semantic accuracy)
- Outperforms autoregressive baselines across ten benchmarks in both zero-shot and fine-tuned settings