Skip to content
Sagan

Paper

Phonemes to the Rescue: Multilingual Tokenization Based on International Phonetic Alphabet

Unreadunread

AI summary

arXiv:2606. 20993v1 Announce Type: new Abstract: Multilingual language models often exhibit performance disparities across languages that can arise as early as the tokenization stage.