The authors propose using topological methods (specifically 0-dimensional persistent homology, which identifies connected components and their lifetimes across scales) to improve LLM alignment. They introduce two techniques: Trajectory Topology Loss (TTL) for supervised fine-tuning, which regularizes the model's update direction to follow "prompt-answer bridges" extracted from the geometry of embeddings, and Topological Preference Optimization (TPO) for DPO, which aligns the improvement direction from rejected to chosen responses with topic-specific semantic vectors. Testing on Qwen2.5-7B-Instruct shows consistent improvements over baselines on preference metrics and LLM-judge evaluations.
Main takeaways:
- Views generation as tracing a trajectory through hidden representation space, not just a sequence of token probabilities
- TTL treats prompt and answer embeddings as a point cloud, uses persistent homology to find topological structure, and aligns model updates with that structure
- TPO constructs semantic preference vectors for topics and guides the DPO update direction in intermediate layers to align with those vectors
- Outperforms non-topological baselines (per-example, nearest-neighbor, random regularizers) on automatic metrics and GPT-judge evaluation
- Maintains or improves toxicity scores while improving preference alignment