The authors built MultiSoc-4D, a 58,000-comment Bengali social-media dataset annotated for category, sentiment, hate speech, and sarcasm, and used it to diagnose a systematic LLM annotation failure they call "instruction-induced label collapse." When asked to label closed-set categories (e.g., hateful/not hateful), ChatGPT, Gemini, Claude, and Grok all showed strong bias toward safe fallback labels ("Other," "Neutral," "No"), missing 79% of hateful content and 75% of sarcasm compared to human-calibrated labels. Even though the models agreed with each other at high rates, Fleiss' kappa was near zero for sarcasm, revealing what the authors call a "label agreement illusion"—models converge on the wrong answer together.
Main takeaways:
- LLMs systematically prefer neutral or "Other" labels in closed-set annotation tasks, especially for minority categories like hate speech and sarcasm.
- High inter-LLM agreement doesn't guarantee quality: the models can all collapse to the same wrong default, producing near-zero kappa despite surface consensus.
- The effect persists across 40+ LLMs of different architectures, suggesting it's a widespread training-pipeline issue rather than model-specific.
- The dataset is released as a diagnostic benchmark for annotation bias in low-resource (Bengali) NLP.
- The bias propagates downstream: models trained on LLM-annotated data inherit the label collapse.