Evaluating literary quality requires assessing interpretive dimensions (cultural representation, emotional depth, philosophical sophistication) that are hard to measure computationally. The authors introduce SAGE, which uses LLMs to evaluate stories across three hierarchical layers (cultural, emotional-psychological, existential-philosophical) with iterative reflection and independent validation. Testing on 100 stories (canonical, pulp fiction, LLM-generated), they achieve 98.8% score convergence and >94% inter-rater agreement. Canonical works consistently outperform pulp and LLM-generated stories on cultural and philosophical dimensions (effect sizes >2.4), but emotional representation shows smaller gaps (d=1.68), suggesting affective patterns are more learnable from training data.
Main takeaways:
- LLM-based evaluation can achieve measurement-grade reliability on complex interpretive dimensions with structured prompts and reflection
- Canonical literature > pulp > LLM-generated on cultural and philosophical depth, with large effect sizes (d>2.4)
- Emotional representation shows smaller gaps (d=1.68), suggesting current LLMs learn affective patterns better than critical stance
- Three quality dimensions (cultural, emotional, philosophical) are empirically distinguishable and moderately correlated (r=0.65-0.68)