The authors introduce GSM-SEM, a framework for generating math-problem variants with high semantic diversity rather than just surface-level perturbations. Most robustness variants of GSM8K swap names or numbers but keep the underlying facts intact; GSM-SEM modifies entities, attributes, and relationships to produce problems that frequently alter the facts and require recomputation, while constraining generation to preserve the original answer and difficulty. Applying GSM-SEM to GSM8K, GSM-Symbolic, and GSM-Plus, they find consistent performance drops across 14 SOTA LLMs (28% average drop at maximum strictness), with larger declines when semantic perturbations combine with symbolic/plus variations. The framework generates fresh variants on each run, reducing reliance on static benchmarks and lowering memorization bias.
Main takeaways:
- GSM-SEM perturbs problem semantics (entities, attributes, relationships) rather than just surface features, often changing underlying facts while preserving the answer.
- Generates fresh variants stochastically on each run, so it's not a fixed benchmark that models can memorize over time.
- Applied to GSM8K, GSM-Symbolic, and GSM-Plus, producing three fully human-validated SEM datasets.
- 14 SOTA LLMs show consistent performance drops (28% average at maximum strictness), especially when semantic + symbolic/plus perturbations combine.
- Demonstrates applicability beyond math by applying GSM-SEM to BigBenchHard, LogicBench, and NLR-BIRD.