Hebatron is a Hebrew-specialized language model built on NVIDIA's sparse Mixture-of-Experts architecture (Nemotron-3). The authors use a three-phase "easy-to-hard" curriculum during training with continuous anti-forgetting anchoring (preventing the model from losing earlier knowledge), then supervised fine-tuning on 2 million bilingual Hebrew-English samples. The curriculum ordering alone yields a 3-point benchmark improvement over training in reverse order. Despite activating only 3B parameters per forward pass (in a 30B total parameter model), Hebatron achieves competitive Hebrew reasoning performance with ~9x higher inference throughput.
Main takeaways:
- First open-weight Hebrew-specialized Mixture-of-Experts model with native long-context support (up to 65,536 tokens).
- Three-phase easy-to-hard curriculum with anti-forgetting anchoring improves Hebrew reasoning by 3 points over reversed training order.
- Achieves 73.8% on Hebrew reasoning benchmarks, outperforming DictaLM-3.0-24B-Thinking (68.9%).
- Only 3B active parameters per forward pass across 30B total, giving ~9x higher throughput than dense models.
- Model weights are released openly for Hebrew and Semitic NLP research.