The author presents TajPersLexon, a curated 40,112-pair Tajik-Persian parallel lexicon for cross-script lexical tasks (retrieval, transliteration, alignment) in a low-resource setting. The benchmark compares three families of methods—lightweight hybrid pipelines, neural sequence-to-sequence, and retrieval—on CPU only. Neural and retrieval baselines hit 98–99% top-1 accuracy, essentially solving exact lexical matching, but large multilingual sentence transformers fail. The hybrid model offers a practical accuracy-efficiency trade-off, achieving 96.4% accuracy on an OCR post-correction task, and is interpretable and fast. All experiments use fixed random seeds for full reproducibility; dataset, code, and models will be released.
Main takeaways:
- TajPersLexon is a 40k-pair Tajik-Persian lexicon for cross-script word/short-phrase matching (Cyrillic Tajik ↔ Persian script).
- Neural seq2seq and retrieval methods nearly solve the task (98–99% top-1 accuracy) on exact lexical matching.
- Multilingual sentence transformers fail despite being large, showing that sentence-level embeddings don't transfer to exact lexical tasks.
- The lightweight hybrid model (rule-based + small neural components) is interpretable, CPU-friendly, and achieves 96.4% on real-world OCR correction.
- All code, data, and models will be open-sourced with fixed seeds for reproducibility.