Translated benchmarks are assumed to inflate scores by preserving English-source cues (the "Translation Tax"), but the authors show this isn't a simple scalar effect when auditing English-to-Chinese benchmarks. Three different measurement approaches disagree: back-translation gaps are small, cue-score calibration doesn't predict item-level gains, and native-control comparisons show model-family effects rather than uniform inflation. An LLM-naturalization stress test (rewriting Chinese surface form while keeping content fixed) reveals item-dependent validity risks rather than a single translation tax—high-residue items benefit from naturalization, low-residue items don't.
Main takeaways:
- The "Translation Tax" (score inflation from English cues in translated benchmarks) isn't a uniform scalar effect
- Three proxy estimators give inconsistent results: back-translation, cue-score calibration, and native-control comparisons
- LLM naturalization (rewriting surface form while preserving content) shows item-dependent effects: some items benefit, others don't
- Translation validity risk depends on estimator choice, item properties, and model family, not a single benchmark-wide factor