Skip to content
Sagan

Paper

Beyond Perplexity: UTF-8 Validity in Byte-aware Language Models

Unreadunread

AI summary

arXiv:2606. 14122v1 Announce Type: new Abstract: Byte-level tokenization enables language models to handle any Unicode input, but models can generate invalid UTF-8 sequences when encountering rare or unseen characters.