The authors present NSMQ Riddles, a benchmark of scientific and mathematical riddles from Ghana's National Science and Maths Quiz, a live TV competition for high school students. Each riddle contains at least three clues (with earlier, vaguer clues fetching more points), and answers are numbers, words, or short phrases, enabling automatic evaluation. The dataset spans 11 years and 1,800 riddles covering biology, chemistry, physics, and math. State-of-the-art LLMs (GPT-4o, Gemini 2.0 Pro, Claude Opus 4, and open models like Kimi-K1.5 and DeepSeek-V3) performed worse than the best student contestants, even in high-reasoning settings, showing the dataset is challenging.
Main takeaways:
- 1,800 riddles from an 11-year archive of Ghana's National Science and Maths Quiz, each with 3+ clues of increasing specificity.
- Answers are short (number, word, or phrase), making automatic evaluation straightforward.
- SOTA closed and open LLMs (tested with high and low reasoning settings) perform worse than top student contestants.
- Addresses underrepresentation of Global South datasets in LLM evaluation.
- Contributes a challenging benchmark for scientific and mathematical reasoning beyond Western educational contexts.