Skip to content
Sagan

Paper

Equity with Efficiency: An Empirical Study of Tokenizers for Multilingual Large Language Models

Unreadunread

AI summary

arXiv:2606. 15044v1 Announce Type: new Abstract: Multilingual large language models (LLMs) depend on subword tokenization to bridge discrete text and continuous neural representation.