Classifying biomedical papers by publication type and study design is important for evidence synthesis, but models trained for high in-domain accuracy often rely on superficial cues (like topic words) rather than true methodological signals, making them brittle under distribution shift. The authors introduce an evaluation framework using controlled semantic perturbations and training strategies (entity masking plus domain-adversarial training) to push models toward relying on explicit methodological language instead of spurious topical correlations. Results show you can improve robustness without sacrificing in-domain accuracy if you selectively suppress non-task-defining features.
Main takeaways:
- Models often classify publication types using topical shortcuts (e.g., "cancer" → observational study) instead of methodology words
- Controlled perturbations (e.g., swapping entity names) reveal this brittleness under distribution shift
- Entity masking + domain-adversarial training forces models to rely on explicit methodological cues
- Robustness and in-domain accuracy can both improve if you selectively suppress spurious features while preserving salient ones